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ABSTRACT 


Traditionally  in  the  Navy/Marine  Corps,  in  an  effort  to  be  proactive  and  prevent 
failures,  maintenance  and  inspections  are  performed  at  fixed  intervals  independent  of 
aircraft  status.  The  current  preventive  maintenance  strategy  services  and  replaces  certain 
components  on  a  predetermined  schedule.  Additionally,  the  current  Navy/Marine  Corps 
aircraft  repair  process  is  reactive.  When  failures  occur,  the  logistics  system  - 
maintenance  and  supply  -  respond.  The  Joint  Strike  Fighter  Autonomic  Logistics  System 
(ALS)  is  proposed  to  be  better  then  the  logistic  system  in  place.  Under  the  ALS 
maintenance  is  performed  only  as  needed.  The  idea  is  to  decrease  the  logistics 
infrastructure  and  simultaneously  improve  logistic  performance,  by  performing 
maintenance  only  as  needed.  Additionally,  parts  are  ordered  ‘autonomously’  without 
human  intervention.  The  logistics  system  prepares  for  an  impending  failure.  In  this  thesis 
simulations  are  developed  to  compare  the  traditional  repair  system  and  the  ALS.  An 
analysis  is  conducted  to  show  differences  in  performance  in  respect  to  aircraft 
availability,  failures  per  mission  and  maintenance-man-hour-per-flight-hour.  The  ALS 
maintenance  model  dominated  traditional  maintenance  under  the  study  assumptions. 
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EXECUTIVE  SUMMARY 


Traditionally  in  the  Navy/Marine  Corps,  to  proactively  prevent  failures, 
maintenance  and  inspections  are  performed  at  fixed  intervals  independent  of  aircraft 
status.  The  current  preventive  maintenance  strategy  services  and  replaces  certain 
components  on  a  predetermined  schedule.  Additionally,  the  current  Navy/Marine  Corps 
aircraft  repair  process  is  reactive.  First  failures  occur,  then  the  logistics  system  - 
maintenance  and  supply  -  respond. 

The  Joint  Strike  Fighter  Autonomic  Logistics  System  (ALS)  is  proposed  be  better 
then  the  logistic  system  in  place.  Under  the  ALS,  maintenance  is  performed  only  as 
needed.  The  idea  is  to  decrease  the  logistics  infrastructure  and  simultaneously  improve 
logistic  performance  by  performing  maintenance  only  as  needed.  Additionally,  parts  are 
ordered  “autonomously”  without  human  intervention.  The  logistics  system  prepares  for 
an  impending  failure. 

Simulations  described  in  this  thesis  compare  the  traditional  repair  system  and  the 
ALS.  An  analysis  shows  the  differences  in  performance  in  respect  to  aircraft  availability, 
failures  per  mission  and  maintenance-man-hour-per-flight-hour. 

Due  to  an  absence  of  JSF  history  and  data  on  JSF  maintenance  processes,  the 
analysis  is  performed  using  an  existing  aircraft.  The  simulations  are  based  on  the  Navy’s 
F/A-18E/F  jet  engine  repair  process.  The  aircraft  will  not  be  replaced  by  the  JSF,  so  it 
may  be  beneficial  to  alter  the  aircraft  in  order  to  take  advantage  of  the  ALS. 

This  thesis  introduces  two  simulations.  One  simulation  is  for  the  traditional, 
F414-GE-400  aircraft  engine  repair  process,  currently  in  place.  The  other  simulation  is 
for  the  E414-GE-400  aircraft  engine  repair  process  under  the  ALS.  Stochastic  models 
developed  are  used  in  a  face  validation  of  the  DES  models.  Design  points  for  the 
simulations  are  selected  using  a  Nearly  Orthogonal-Eatin  Hypercube  (NOEH)  design. 
Regression  models  are  produced  to  define  the  relationship  between  each  of  the  MOEs 
and  the  predictor  variables.  The  EPM  regression  models  are  used  to  compare  the 
traditional  repair  system  and  the  ALS. 


XXI 


The  ALS  maintenance  model  dominates  in  terms  of  flights  per  mission  (FPM)  and 
maintenance  man  hour  per  mission  (MMHF).  However,  large  gains  in  operational 
availability  were  not  realized.  In  terms  of  FPM,  when  deciding  between  investing  in 
module  reliability  in  the  traditional  repair  system  or  prognostic  accuracy  it  is  best  to 
invest  in  prognostics  and  switch  to  the  ALS.  In  terms  of  MMHF,  the  ALS  potential  far 
exceeds  that  of  the  traditional  system.  The  ALS  is  superior  to  the  traditional  system. 


I.  INTRODUCTION 


A.  BACKGROUND/AREA  OF  RESEARCH 

1.  Introduction 

In  November  2002,  Department  of  Defense  (DoD)  policy  directed  the  military 
services  to  implement  the  tenets  of  Conditioned  Based  Maintenance  Plus  (CBM+)  in 
weapons  systems  and  logistics  support  programs  where  cost  effective.  CBM+  focuses  on 
predicting  maintenance  needs  and  responding  accordingly.  The  idea  is  to  decrease  the 
logistics  infrastructure  and  simultaneously  improve  logistic  performance  by  performing 
maintenance  only  as  needed.  “The  Joint  Strike  Fighter  (JSF)  Prognostics  Health 
Management  System  (PHM)  is  highlighted  as  an  “emerging”  example  of  a  true 
prognostics-capable  aircraft  embodying  the  full  intent  of  the  CBM-H”  (Smith,  2003). 

Prognostic  capability  alone  does  not  improve  the  entire  maintenance  and  logistics 
system.  “Prognostic  capability  may  properly  identify  a  material  requirement,  but  the 
requirement  must  be  transmitted,  received,  fdled,  transported,  and  delivered  to  provide  a 
solution  to  the  maintainer”  (DoD,  2004).  A  logistic  network  where  real  time 
maintenance  and  supply  needs  are  known  is  essential  for  faster  support.  This  is  in 
agreement  with  the  “sense  and  respond  logistics”  concept.  The  customer  has  only  what 
they  need  when  they  need  it.  For  this  to  be  viable,  “logisticians  need  to  be  able  to 
communicate  with  customers,  to  know  where  they  are  geographically,  to  know  what  they 
are  doing...”(Schrady,  2005). 

To  take  advantage  of  its  prognostic  capability,  and  to  accelerate  the  logistics 
process,  the  JSF  PHM  comes  as  part  of  an  Autonomic  Logistics  System  (ALS)  that 
includes  a  distributed  information  system.  According  to  the  JSF  website:  The  JSF  will 
achieve  unprecedented  levels  of  reliability  and  maintainability,  will  be  the  most 
supportable  aircraft,  and  will  be  ready  to  fight  anytime  and  anyplace  (F-35,  2006). 

2.  Preventive  Maintenance/Reactive  Support 

Traditionally  in  the  Navy/Marine  Corps,  to  proactively  prevent  failures, 
maintenance  and  inspections  are  performed  at  fixed  intervals  independent  of  aircraft 
status.  The  current  preventive  maintenance  strategy  services  and  replaces  certain 
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components  on  a  pre-determined  schedule.  This  periodic  maintenance  depends  strictly  on 
flight  hours  accumulated  on  the  component  (age).  Preventive  maintenance  is 
independent  of  the  component’s  condition.  Scheduled/preventive  maintenance  wastes 
time  and  money;  some  of  the  scheduled  maintenance  is  unnecessary.  Only  23%  of 
aircraft  equipment  failure  patterns  are  age  related,  77%  are  random  (Sondalini,  2003). 
For  random  failures,  one  cannot  predict  when  a  failure  occurs  based  only  on  age. 

Additionally,  current  logistics  and  maintenance  support  are  reactive.  First  failures 
occur,  then  the  logistics  system  -  maintenance  and  supply  -  respond.  The  maintenance 
personnel  troubleshoot  to  isolate  the  problem,  then  order  the  part  and  wait  for  supply  to 
requisition  the  request.  Time  is  wasted  while  waiting  on  parts.  The  Automated  Logistics 
system  (ALS)  remedies  these  two  areas:  scheduled  maintenance  based  on  age  and 
responsiveness  to  failure. 

3.  Autonomic  Logistics  System 

An  Autonomic  Logistics  System  (ALS)  is  a  system  that  automatically  responds  to 
maintenance/failure  events.  Tasks  such  as  identifying  failures  and  ordering  parts  are 
executed  without  human  intervention.  “The  JSF  Autonomic  logistics  system 
encompasses  three  essential  components:  (1)  a  highly  reliable,  maintainable,  and 
intelligent  aircraft  that  incorporates  Prognostics  Health  Management  (PHM)  technology; 
(2)  a  technologically-enabled  maintainer,  capable  of  effectively  and  efficiently 
maintaining  the  JSF;  and  (3)  a  Joint  Distributed  Information  System  (JDIS)  that 
incorporates  advanced  information  system  technology  to  provide  decision  support  tools 
and  an  effective  communication  network  linking  the  JSF  with  the  logistics  infrastructure” 
(Hess,  2004), 

The  key  to  the  JSF  Autonomic  Logistics  is  the  ability  to  detect  and  predict 
failures.  These  capabilities  in  conjunction  with  the  other  essential  components  not  only 
have  the  potential  of  saving  time  but  make  a  policy  of  condition-based  maintenance  a 
realistic  one.  This  way  maintenance  is  performed  only  as  needed.  Another  benefit  to 
predicting  failures  is  to  stop  disabling  failures  and  fatal  failures  before  they  occur. 
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The  PHM  technology  incorporates: 

•  Diagnostics  -  the  process  of  determining  the  state  of  a  component  to 
perform  its  function(s) 

•  Prognostics  -  predictive  diagnostics  which  includes  determining  the 
remaining  life  or  time  span  of  proper  operation  of  a  component 

•  Health  Management  -  the  capability  to  make  appropriate  decisions  about 
maintenance  actions  based  on  diagnostics/prognostics  information, 
available  resources  and  operational  demand”  (Hess,  2006). 

Ideally  the  PHM  detects/predicts  a  component  failure  and  instantaneously  relays 
this  immediately  to  the  appropriate  entities.  The  ALS  automatically  provides  details  of 
the  cause  and  location  of  the  failure;  troubleshooting  is  eliminated.  Additionally, 
required  parts  are  ordered  automatically  eliminating  the  requirement  to  manually  order 
parts  and  reducing  the  time  for  a  part  to  arrive.  The  ALS  will  provide  the  required 
number  of  personnel  to  repair  the  failure  and  will  locate  available  parts.  Knowing  a 
failure  will  occur  in  advance  allows  additional  time  for  preparation  and  flexibility  in 
scheduling  maintenance.  For  example,  if  a  failure  was  detected  30  hours  in  advance,  the 
squadron  may  decide  to  continue  using  the  aircraft  until  the  parts  arrived.  See  Figure  1 
for  overview  of  process. 


3 


A/C  -  aircraft 

UP  -  A/C  is  mission  capable,  can  fly,  can  be  scheduled  for  a  mission 

DOWN  -  A/C  non-mission  capable,  cannot  fly,  cannot  be  scheduled  for  a  mission 


Figure  1.  JSF  Autonomic  Logistic  Flow  Chart 
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4,  Assumed  Benefits  of  the  ALS 

By  predicting  failures  and  accelerating  information  flow  the  ALS  has  potential  to 
decrease  the  downtime  of  the  aircraft  and  to  reduce  cost.  A  financial  advantage  may  exist 
despite  the  fact  there  is  an  expense  in  setting  up  and  maintaining  the  ALS.  An 
Army/Marine  CBM+  initiated  program  to  develop  prognostics  on  helicopters  has  proven 
to  increase  readiness.  However,  few  helicopters  are  installed  with  the  equipment,  as  costs 
exceed  $1  million  dollars  per  aircraft  (Messenger,  2004). 

The  expectations  for  the  ALS  are  high.  It  is  anticipated  there  will  be  a  forty  to 
fifty  percent  savings  in  manpower  and  infrastructure  cost  at  unit  level  while 
simultaneously  turn  around  time  will  be  in  minutes  instead  of  hours  (Adams,  2003). 
Supposedly,  aircraft  readiness  will  increase,  down  time  for  repairs  will  decrease,  required 
number  of  maintenance  personnel  will  decrease,  and  reducing  cost. 

Dr.  Scheuren  heads  the  Defense  Advanced  Research  Projects  Agency’s  (DAPRA) 
Joint  Advanced  Strike  Technologies  (JAST)  Program.  In  his  view,  “both  the  ultimate 
objective  of  PHM  and  its  related  autonomic  logistics  system  is  to  reduce  maintenance 
manpower  requirements  by  approximately  20%  to  40%,  increase  combat  sorties  by  25%, 
and  reduce  the  complexity  of  the  logistics  trail  by  50%,  compared  to  current  military 
strike  aircraft;  all  at  a  cheaper  life  cycle  cost  as  compared  with  legacy  aircraft” 
(Nickerson,  1998). 

Rebulanan  constructed  a  simulation  of  the  basic  framework  of  an  ALS  (ALSim) 
as  a  tool  to  allow  comparison  between  ALS  and  the  current  maintenance  process 
(Rebulanan,  2000).  His  model  showed  that  higher  aircraft  availability  could  be  obtained 
with  an  ALS.  Rebulanan  assumed  the  prognostics  were  100%  accurate  and  maintainers 
were  always  available.  One  would  expect  ALS  with  100%  accurate  prognostics  to 
perform  better  than  the  current  system.  What  if  the  prognostics  were  not  100%?  In  this 
case,  false  detections  and/or  missed  detections  are  possible.  With  a  false  detection,  the 
logistics  system  is  unnecessarily  burdened.  Advantages  and  disadvantages  for  the  current 
and  ALS  processes  are  summarized  below. 
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Current  System; 


Advantages 

•  Scheduled/Preventive  maintenance  may  prevent  catastrophic  failure,  thus 
exposing  the  underlying  cause  of  such  a  failure.  Under  ALS  these  causes 
may  be  masked. 

Disadvantages 

•  Fault  occurs,  then  maintenance  must  wait  for  aircraft  to  land  to  determine 
problem. 

•  High  time  to  troubleshoot  aircraft  (includes  diagnostics,  it  takes  time  to 
retrieve  information) 

•  Time  taken  to  order  part  and  wait  for  delivery  (aircraft  may  be  unusable). 

•  Wasted  resources:  scheduled  maintenance  replaces  parts  that  may  not  need 
replacing.  Other  maintenance  performed  that  may  not  be  required. 

•  Wasted  time:  unnecessary  scheduled  maintenance  takes  time. 

ALS  system; 

Advantages 

•  Fault  detected  before  it  occurs,  and  information  is  relayed  instantaneously. 

•  Parts  are  automatically  ordered,  saving  time. 

•  No  troubleshooting;  therefore  saves  time. 

•  Can  plan  maintenance  for  aircraft  at  ideal  time,  i.e.  when  part  is  delivered. 

•  No  scheduled/preventive  maintenance  saving  man-hours,  cost  and 
increasing  readiness. 

Disadvantages 

•  False  alarms  may  cause  unnecessary  part  replacement  and  maintenance. 

•  Chance  of  not  detecting  a  failure. 

•  The  time  from  detection  of  a  degraded  component  until  failure  may  be  too 
short  or  too  variable  to  gain  benefits. 

•  Cost  of  sensors,  communication  equipment,  etc. 


B.  RESEARCH  OBJECTIVES  /  PURPOSE 

The  JSF  ALS  promises  to  perform  better  than  compared  to  the  current  system. 
The  intent  of  the  thesis  is  to  examine  how  the  ALS  performs.  The  ALS  automatically 
orders  parts  based  on  the  PHM.  For  a  PHM  with  various  degrees  of  accuracy,  failure 
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detection  times  and  false  alarm  rates  -  including  JSF  ORD  (operational  requirement 
document)  requirements  -  determine  the  logistic  implications,  e.g.  parts  ordered  in  error 
that  were  not  needed.  Indicators  of  how  well  the  logistic  system  is  doing  include:  time  for 
maintenance  to  receive  a  part,  time  aircraft  is  down  and  number  of  components  replaced 
needlessly.  Additionally,  this  thesis  estimates  the  maintenance  time  saved  compared  to 
using  current  policies  and  the  difference  in  aircraft  readiness. 

Critical  factors  for  the  traditional  repair  system  and  the  ALS  are  identified.  The 
factors  are  used  to  compare  both  systems.  The  purpose  of  this  thesis  is  to  provide 
analytic  support  for  the  hypothetical  benefits  of  the  ALS  over  the  current  system.  The 
model  will  aid  in  determining  how  the  ALS  will  perform,  and  therefore,  help  decision 
makers  determine  how  much  to  invest. 

C.  SCOPE  AND  ASSUMPTIONS 

The  current  logistics  structure  with  the  ALS  is  shown  in  Figure  2.  The  JDIS 
component  is  assumed  to  work  perfectly,  and  so  will  not  be  considered.  If  the  PHM  on 
board  predicts  a  failure,  a  signal  is  generated.  Signals  from  the  aircraft  are  assumed 
instantaneously  received,  without  interference.  Information  is  available  real  time.  The 
signal  initiates  what  parts  are  ordered  and  the  maintenance  actions  needed. 

The  ALS  still  requires  human  involvement  to  accomplish  tasks.  As  an  example 
parts  are  ordered  autonomously  but  a  supply  clerk  is  needed  to  fdl  the  requisition. 
Therefore,  there  may  be  delay  for  a  human  to  respond  to  the  message.  The  autonomic 
driven  actions  respond  immediately  to  the  signal.  Human  response  to  the  signal  depends 
on  availability  and  capability  of  personnel.  However,  in  this  study,  it  is  assumed  that 
capable  personnel  are  always  available. 

Prognostics  are  not  presumed  to  be  100%  correct/ac curate.  As  a  result  one  of  the 
following  scenarios  may  occur: 

(1)  PHM  accurately  predicted  the  failure. 

(2)  False  alarm:  PHM  sends  signal  when  no  failure  is  to  occur. 

(3)  PHM  does  not  predict/send  signal  before  the  failure  actually  occurs. 
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Additional  Assumptions; 

(1)  Diagnostics  are  100  %  accurate.  Once  a  failure  is  predicted,  the  fault  is 
accurately  isolated. 

(2)  The  manufaeturer/depot  has  unlimited  resources  and  replaees  all  non¬ 
ready  for  issue  (NRFI)  line  replaceable  units  (LRU)  immediately. 


Signal  sent  from  aircraft  (A/C)  ^ 

Path  of  RFI  (ready  for  issue)  part 
Path  of  NRFI  (Not  ready  for  issue)  part 
Squadron  (SQUAD)  where  maintenance  performed 


Figure  2.  JSF  Autonomic  Logistic  Structure 


D,  PRIOR  WORK 

In  2000,  Capt.  Rene  Rebulanan,  United  States  Air  force,  wrote  a  thesis  to  examine 
how  the  ALS  performs  and  what  demands  it  places  on  the  logistics  infrastructure 
(Rebulanan  2000).  Using  the  Java  programming  language  he  developed  an  ALS 
simulation  model.  For  a  set  of  four  aircraft  he  showed  the  difference  in  flying  the  aircraft 
with  ALS  and  without  ALS  (the  traditional  system).  Aircraft  characteristics  were  based 
on  the  F-16.  The  simulation  was  run  for  to  simulate  six  months.  Rebulanan  concluded 
that  the  ALS  leads  to  higher  availability.  Rebulanan  assumed  the  prognostics  worked  all 
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the  time,  that  is  impending  failures  were  always  deteeted.  This  thesis  explores  the  impaet 
of  prognosties  that  are  not  100%  aeeurate.  Additionally,  this  thesis  implements  a 
squadron  worth  of  aireraft,  and  analyses  the  impaet  on  the  O  and  I  levels  over  the  lifetime 
of  the  aireraft  (assumed  to  be  25  years). 

In  2003,  Lieutenant  Commander,  Erie  J.  Sehoeh,  wrote  a  thesis  that  models  the 
F414-GE-400  engine  repair  proeess.  The  goal  of  the  simulation  was  to  provide 
operational  availability  and  probability  to  spare  the  proeess  given  the  eurrent  system  in 
plaee,  whieh  does  not  ineorporate  the  AES.  His  simulation  utilizes  the  Java  paekage 
Simkit,  a  software  paekage  for  implementing  Diserete  Event  Simulation  (DES)  models 
(Buss,  2001).  Sehoeh’s  model  aeeounts  for  all  the  squadrons  and  AIMDs  in  the  United 
States  Navy.  This  thesis  will  implement  only  one  AIMD  and  the  squadrons  it  supports. 
Sehoeh’ s  model  mimics  the  depot  with  same  level  of  detail  as  the  AIMD,  and  depot 
inventory  levels  and  turn  around  times  are  considered.  The  logistic  impact  on  the  depot 
falls  outside  the  scope  of  the  thesis.  The  depot  is  assumed  to  have  a  replacement  on  hand. 
This  thesis  develops  a  simulation  for  the  F414-GE-400  repair  process  augmented  with 
AES. 

E,  METHODOLOGY 

Because  of  the  absence  of  history  and  data  on  the  JSE  maintenance  processes,  the 
analysis  will  be  performed  using  an  existing  aircraft.  The  model  is  based  on  the  Navy’s 
F/A-18E/F  jet  engine  repair  process.  Therefore,  model  runs  are  based  on  the  Naval 
Aviation  Maintenance  Program  (NAMP).  Two  F-414-GE-400  engines  power  the  F/A- 
18E/F.  The  line  replacement  units  (ERU)  simulated  are  the  F-414-GE-400  engine  and  its 
subcomponents. 

A  Discrete  Event  Simulation  (DES)  model  represents  the  current  F-414-GE-400 
engine  repair  process.  With  a  DES  model  one  can  observe  the  behavior  of  the  system 
over  time.  The  DES  program  is  based  on  Lieutenant  Commander  Schoch’s  simulation 
work  so  it  is  also  implemented  in  Simkit  (Sehoeh,  2003).  Multiple  runs  (with  multiple 
replications)  for  the  current  system  model  are  used.  The  effect  of  changing  the  input 
parameters  on  the  measures  of  effectiveness  (MOE)  is  explored. 
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The  model  will  then  be  augmented  with  the  essential  elements  of  the  ALS:  PHM 
and  JDIS.  The  ALS  model  is  run  multiple  times  using  various  prognostie  accuracies, 
detection  times  and  false  alarm  rates.  Example  ranged  include  prognostic  accuracies:  90, 
95  and  100%,  detection  times:  10,  20  and  30  hours  before  failure  time  and  mean  flight 
hours  between  false  alarms  450  and  700.  The  same  LRU  failure  patterns  for  both  the 
legacy  system  and  the  ALS  are  used.  For  each  level  of  accuracy,  expected  different 
values  of  the  MOE  are  observed.  For  some  level  of  accuracy  and  below  the  ALS  may 
offer  little  or  no  improvement. 

Simulation  results  for  each  system  are  compared  to  determine  the  benefits  of  the 
ALS.  Three  MOEs  are  considered.  The  primary  MOE  is  operational  availability:  time  the 
aircraft  are  available  for  a  mission  divided  by  the  total  time  of  the  simulation  run.  The 
other  MOEs  considered  are  maintenance  man  hours  per  flight  hour  (MMHF)  and  number 
of failures  per  flight  mission  (FPM). 

Additionally,  a  stochastic  model  is  developed  for  each  system.  The  purpose  of 
the  stochastic  model  is  to  analyze  key  elements  of  the  logistics  system.  The  results  from 
the  stochastic  model  are  a  face-validation  of  the  DES  model. 

The  thesis  consists  of  five  steps: 

1.  Create  a  Discrete  Event  Simulation  (DES)  using  Simkit  for  the  current 
F414-GE-400  engine  repair  process. 

2.  Augment  the  current  F414-GE-400  engine  repair  process  model  with  ALS. 

3.  Develop  stochastic  models  and  use  them  to  verify  the  DES  model. 

4.  Exercise  the  DES  model  using  F/A-18E/F  data. 

5.  Analyze  results  of  simulation  and  draw  conclusions. 

The  analysis  shows  that  ALS  maintenance  model  dominates  in  terms  of  flights  per 
mission  (FPM)  and  maintenance  man  hour  per  mission  (MMHF).  However,  large  gains  in 
operational  availability  were  not  realized.  In  terms  of  FPM,  when  deciding  between 
investing  in  module  reliability  in  the  traditional  repair  system  or  prognostic  accuracy  it  is 
best  to  invest  in  prognostics  and  switch  to  the  ALS.  In  terms  of  MMHF,  the  ALS 
potential  far  exceeds  that  of  the  traditional  system.  Thus,  the  analysis  supports  the 
conclusion  that  ALS  is  superior  to  the  traditional  system. 
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F.  ORGANIZATION 

Chapter  II,  Logistics  Process,  describes  the  current  F4I4-GE-400  aircraft  engine 
repair  and  the  F4I4-GE-400  engine  repair  with  AES  processes.  Additionally  Chapter  II 
covers  data  used  to  model  the  engine  repair  process.  Chapter  III  describes  the  DES 
model.  Chapter  IV  contains  the  stochastic  models.  In  this  chapter,  results  from 
simulations  using  the  DES  and  stochastic  models  are  compared.  Chapter  V  discusses  the 
experimental  design.  Chapter  VI  is  the  analysis  of  the  MOEs.  Finally,  Chapter  VII 
provides  conclusions  and  further  recommendations. 
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II.  F414-GE-400  AIRCRAFT  ENGINE  REPAIR  PROCESS 


Due  to  the  absenee  of  history  and  data  on  the  JSF  engine  maintenanee  processes, 
the  F/A-18E/F  ‘Super  Hornet’  engine,  F-414-GE-400,  is  used  for  this  study.  The  E/A  - 
18E/E  is  the  latest  tactical  aircraft  to  enter  the  United  States  Navy.  It  will  not  be  replaced 
by  the  JSE  so  it  may  be  beneficial  to  alter  the  aircraft  in  order  to  take  advantage  of  the 
AES.  The  purpose  of  this  chapter  is  to  describe  the  E414-GE-400  Aircraft  Engine  Repair 
Process  and  give  an  overview  of  the  data  and  assumptions. 

A,  F-414-GE-400  ENGINE 

Two  engines  power  the  E/A-18E/E.  The  E-414-GE-400  consists  primarily  of  six 
modules:  The  fan,  compressor,  combustor,  high  pressure  turbine,  low  pressure  turbine 
and  afterburner.  The  engine  is  designed  for  easier  more  efficient  maintenance  and  the 
modules  are  fully  interchangeable. 


Combostof 


High  Pressure  Low  Pressure 
Turbine  Turbine 


Eigure  3.  E414-GE-400  (General  Electric,  2003) 


B.  THREE  LEVELS  OF  MAINTENANCE 

The  Naval  Aviation  Maintenance  Program  (NAMP)  is  founded  upon  the  three- 
level  maintenance  concept:  Organizational  (O),  Intermediate  (I)  and  Depot  (D).  As  the 
operating  unit,  the  squadron  performs  the  lowest  level  of  maintenance:  organizational 
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maintenance.  The  Aviation  Intermediate  Maintenance  Depot  (AIMD),  located  at  air 
stations,  as  well  as  deployed  aircraft  carriers,  performs  the  intermediate  level 
maintenance.  The  naval  aviation  industrial  establishment  performs  the  depot  level 
maintenance.  For  the  FA-18E/F,  the  engine  repair  process  is  identical  while  afloat  or 
onshore. 

The  squadron  performs  maintenance  to  support  its  daily  operations.  The  squadron 
performs  limited  repair,  which  includes  engine  trouble  shooting,  and  removal  and 
installation.  The  engine  is  removed  when  a  failure  that  cannot  be  repaired  occurs  or 
when  one  its  modules  is  scheduled  for  replacement. 

Traditionally,  AIMD  has  extensive  maintenance  capabilities.  However,  I-level 
support  for  the  F14-GE-400  is  limited.  Eimited  I-level  support  is  also  planned  for  the 
JSE.  The  AIMD  has  the  ability  exchange  modules  on  an  engine  and  assemble  engines. 
The  AIMD  does  not  repair  modules.  Modules  in  need  of  repair  or  preventive 
maintenance  are  sent  to  a  naval  aviation  industrial  establishment. 

Eor  the  E414-GE-400,  there  is  only  one  naval  aviation  industrial  establishment, 
the  Naval  Aviation  Depot  at  Jacksonville,  Elorida.  “The  D-level  is  the  top  echelon  of  the 
jet  aircraft  engine  repair  process  and  can  perform  all  maintenance  and  repair  action” 
(Schoch,  2003). 


C.  F/A18-E/F  ENGINE  REPAIR  CYCLE 

When  an  engine  failure  is  detected,  the  squadron  troubleshoots  to  isolate  the  cause 
of  failure.  The  engine  is  removed  if  the  failure  cannot  be  repaired  at  the  0-level. 
Additionally,  engines  are  removed  if  any  of  the  modules  has  reached  ‘high  time.’  High 
time  is  a  predetermined  time  when  the  module  is  scheduled  for  removal.  High  times 
prescribe  the  removal  of  modules  based  on  hours  of  operation.  When  a  module  reaches 
high  time,  it  is  removed  regardless  of  its  current  condition.  Module  high  times  are  listed 
in  Appendix  A.  Figure  4  illustrates  the  0-level  echelon  maintenance. 
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Figure  4.  0-level  F414-GE-400  Engine  Repair 

Managing  an  engine  pool  and  a  module  pool  is  a  key  function  of  the  AIMD.  The 
AIMD  has  the  ability  to  disassemble  and  assemble  engines.  Modules  that  need  repair,  to 
include  preventive  maintenance,  are  sent  to  the  depot.  A  squadron  that  needs  an  engine 
receives  a  ready  for  issue  (REI)  engine  from  the  pool.  As  long  as  there  is  an  engine  in  the 
inventory,  the  squadron  does  not  have  to  wait  for  the  removed  engine  to  be  repaired. 
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AIMD  inspects  all  engines  received  from  the  squadrons.  Modules  in  need  of 
repair  or  preventive  maintenance  are  replaced.  AIMD  has  the  option  of  replacing 
modules  with  modules  in  the  pool  or  usable  modules  from  other  engines  waiting 
assembly.  As  engines  are  assembled,  they  are  added  to  the  pool. 

At  the  I-level,  a  module  is  considered  in  need  of  preventive  maintenance  if  it  has 
reached  high  time  or  if  it  is  within  a  certain  range  of  high  time.  This  range  of  time  is 
called  ‘build  window.’  For  example,  if  an  engine  with  a  build  window  of  twenty  hours  is 
sent  to  the  AIMD,  modules  within  twenty  hours  of  high  time  are  replaced.  Figure  5 
illustrates  the  I-level  echelon  maintenance. 
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Figure  5.  I-level  F414-GE-400  Engine  Repair 


The  Naval  Aviation  Depot  at  Jaeksonville,  Florida  repairs  modules.  It  is  the 
responsibility  of  the  Depot  to  replaee  modules  turned  in  from  the  AIMD.  Eike  the 
AIMD,  the  depot  has  an  inventory  pool  in  whieh  to  issue  modules. 

D.  ALS  F/A18-E/F  ENGINE  REPAIR  CYCLE 

Under  the  ALS,  aireraft  are  enabled  with  the  PHM  system.  The  system  monitors 

the  eurrent  eondition  of  the  engine  and  foreeasts  impending  failures.  In  flight,  when  the 
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PHM  detects  any  degradation  of  the  engine,  it  will  isolate  the  failure  and  predict  the 
remaining  life  of  the  engine.  Note  that  the  PHM  is  not  100%  accurate.  False  detections 
and/or  missed  detections  may  occur.  A  signal  containing  this  information  is  sent  from 
the  aircraft  to  each  of  the  three  levels  of  maintenance  thereby  autonomously  initiating 
action  at  each  level. 

The  squadron  does  not  troubleshoot  engines  nor  does  it  remove  modules  on  the 
high  time  schedule.  Upon  receiving  an  impending  failure  message  the  squadron  will 
decide  when  to  remove  the  engine  from  the  aircraft.  Ideally,  the  engine  removal  can 
occur  to  coincide  with  the  arrival  of  the  replacement  engine.  Aircraft  will  no  longer  sit 
inoperable  while  waiting  for  a  replacement.  The  longer  the  lead  time  (predicted 
remaining  life  of  engine)  the  more  flexibility  the  squadron  has.  The  squadron  has  two 
options:  to  abort  the  mission  then  ground  the  aircraft,  or  not  ground  the  aircraft  and 
schedule  the  aircraft  for  follow  on  missions.  Figure  6  illustrates  the  0-level  echelon 
maintenance  at  the  squadron. 


Figure  6.  ALS  0-level  F414-GE-400  Engine  Repair 

When  an  impending  failure  message  from  the  aircraft,  is  received  at  the  I-level,  an 
engine  is  sent  to  the  squadron.  The  degraded  engine  is  brought  back  to  the  AIMD.  The 


18 


AIMD  no  longer  performs  inspeetions.  The  PHM  directs  replacement  of  the  module  that 
caused  the  failure.  The  degraded  module  is  then  sent  to  the  depot.  The  depot 
automatically  sends  a  RFI  module  to  the  AIMD  and  waits  for  the  degraded  module. 
Figure  7  illustrates  the  I-level  echelon  maintenance. 


Figure  7.  ALS  I-level  F414-GE-400  Engine  Repair 
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E,  ASSUMPTIONS 

1.  Assumptions  Applicable  from  Schoch’s  Thesis 

The  simulation  uses  the  following  assumptions  from  Sehoeh’s  thesis  (Schoeh, 

2003): 

•  All  levels  of  repair  operate  24  hours  a  day. 

•  A  year,  for  the  purpose  of  this  simulation,  is  exaetly  52  weeks  long. 

•  Engines  and  modules  are  always  repaired  and  never  eondemned. 

•  The  D-Level  repairs  all  modules,  the  I-  Level  does  not. 

•  The  I-Level  assembles  all  engines,  the  D-  Level  does  not. 

•  Engine  repair  at  the  I-Level  begins  as  soon  as  there  are  enough  modules  to 
eomplete  an  engine. 

•  The  0-Level  has  unlimited  engine  removal  and  installation  eapaeity. 

•  The  shipping  times  between  the  I  and  D-Levels  are  a  eonstant. 

•  The  0-Level  removes  an  engine  from  a  L/A-18  only  for  module  failures 
and  high  times.  All  other  maintenanee  requirements  have  a  negligible 
effect  on  overall  operational  availability. 

•  L414-GE-400  engine  failures  are  the  result  of  independent  failures  of  the 
modules  in  the  engine. 

•  Once  an  L/A-18’s  flight  schedule  is  set,  the  L/A-18  does  not  vary  from  it 
(except  in  the  case  of  engine  failure  or  high  time  because  it  is  not  flyable). 

•  If  a  L/A-18  experiences  an  engine  failure,  it  lands  immediately.  No 
emergency  diverts,  aircraft  lands  back  at  base.  The  model  assumes  no 
return  to  origin  delay. 

•  A  L/A-18  may  have  both  engines  fail  simultaneously. 

2.  Additional  Assumptions 

The  simulation  uses  the  following  additional  assumptions: 

•  The  depot  is  assumed  to  have  unlimited  resources  and  therefore  always 
has  modules  on  hand  ready  for  issue. 

•  The  intricacies  of  the  depot  (e.g.  man-hours)  are  not  analyzed. 

•  There  has  been  a  recent  push  to  privatize  the  Depot. 

•  Lockheed  Martin  will  provide  depot  level  support  for  the  JSL  under  a 
Performance-Based  Logistics  (PBL)  agreement. 

•  Because,  a  PBL  agreement  is  to  guarantee  part  delivery  within  a  certain 
time,  it  is  assumed  assigning  various  values  to  the  shipping  time  between 
the  depot  and  the  AIMDs  captures  the  impact  of  depot  level  support. 
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•  Upon  repair,  modules  are  good  as  new. 

•  For  the  ALS,  an  F/A-18’s  flight  sehedule  is  set,  the  F/A-18  does  not  vary 
from  it  (exeept  in  the  ease  of  deteeted  pending  engine  failure  or  engine 
failure  beeause  it  is  not  flyable). 

•  Onee  an  RFI  engine  is  installed  in  an  aireraft,  it  is  eonsidered  ready  for 
normal  flight.  Funetional  cheek  flights  are  not  required. 

•  Cannibalization,  taking  parts  off  one  aircraft  to  install  in  another  is  not  an 
option. 

•  All  aircraft  enter  service  at  the  same  time. 

•  All  flights  scheduled  for  2.75  hours. 

F.  DATA 

1.  Data  Source 

The  following  data/values  were  obtained  from  Lieutenant  Commander  Schoch’s 
(Schoch,  2003)  thesis:  module  failure  data,  module  high  times,  O  and  I  level  module 
engine  removal,  installation  and  inspection  times,  time  to  transfer  engine  from  AIMD  to 
squadron,  number  of  squadrons  and  aircraft,  and  engine,  module  AIMD  allowance.  All 
values,  except  the  module  failure  data,  are  listed  in  Appendix  A. 

2,  Data  Analysis  of  Module  Time  Between  Failures 

Text  files  containing  time  between  failures  data  for  each  module  type  were 
analyzed.  Using  IMP  statistical  software,  the  Exponential,  Weibull,  Gamma,  Normal, 
LogNormal  and  Beta  distributions  were  fit  to  each  module  failure  data.  The  Beta 
distribution  is  the  best  fit  for  each  module  type.  The  goodness-of-fit  test  yielded  the 
following  p-values: 

•  Module  1  (Fan)  =  0.2500 

•  Module  2  (Compressor)  =  0.2500 

•  Module  3  (Combustor)  =  0.1820 

•  Module  4  (HPT)  =  0.2500 

•  Module  5  (LPT)  =  0.2500 

•  Module  6  (After  Burner)  =  0.2500 

For  each  module  type,  evidence  against  the  null  hypothesis:  Data  is  Beta 
distributed,  is  not  significant. 
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G.  CONCLUSION 

The  preceding  section  described  the  current,  traditional  logistic  system  which 
facilitates  the  F-414-GE-400  engine  repair  process.  Then  the  F-414-GE-400  repair 
process  was  described  under  the  AES.  The  key  components  necessary  to  model  the 
system  are  identified.  The  traditional  logistic  system  and  the  AES  are  different.  The 
components  and  assumptions  mentioned  in  this  chapter  are  used  to  model  the  systems. 
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III.  DISCRETE  EVENT  SIMULATION  (DES)  MODEL 


Simkit  event  graphs  aid  in  describing  the  simulation  models  for  both  the 
traditional  and  ALS  systems.  Event  graphs  also  highlight  the  difference  between  the  two 
models. 

A,  DES,  EVENT  GRAPHS  AND  SIMKIT 

1,  DES  Paradigm 

DES  is  a  modeling  paradigm  in  which  the  model’s  state  remains  constant  except 
for  particular  events,  which  can  take  place  at  any  place  or  time  (Eaw  and  Kelton,  2000). 
Events  are  state  transitions.  The  state  of  a  system  is  the  collection  of  variables  necessary 
to  describe  the  status  of  the  system  at  any  given  time  (Winston,  2004).  The  state  changes 
at  discrete  points  in  time.  Scheduled  events  are  placed  in  a  Future  Event  List  (FEE)  in 
chronological  order.  During  the  simulation,  time  is  advanced  in  discrete  steps  to  the  next 
earliest  event  in  the  FEE. 

2,  Event  Graphs 

Event  graphs  are  diagrams  of  physically  representing  discrete  event  simulation 
models  (Buss,  2002).  Events  are  represented  by  nodes.  Directed  edges  are  the 
scheduling  relationship  between  pair  of  nodes.  Nodes  are  the  state  transitions  and  arcs 
perform  the  scheduling.  For  the  diagram  below,  if  (i  ==  true)  then  the  occurrence  of  A 
causes  the  scheduling  of  B  to  occur  after  a  delay  of  time  t. 
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3,  Listener  Pattern 

A  useful  application  in  DBS  is  the  ability  to  make  simulated  components  “listen” 
to  another’s  events.  When  a  scheduled  event  of  the  source  component  occurs,  the  event 
is  simultaneously  scheduled  for  all  components  listening  to  the  source  with  the  same 
event.  In  Figure  9  below,  class  B  listens  to  class  A.  The  arrival  method  in  class  B  is 
scheduled  by  the  self-scheduling  arrival  method  of  class  A.  For  more  information  on  DBS 
and  Simkit,  see  the  reference  (Buss,  2002). 


CLASS  A  CLASS  B 


4,  SimKit 

All  models  in  this  chapter  have  been  developed  using  Simkit.  Simkit  is  a  DBS 
implementation  software  package  written  in  Java,  which  directly  supports  building  event 
graph  models.  Bvery  event  in  an  event  graph  model  corresponds  to  an  instance  method  in 
a  Simkit  class.  The  relationship  between  Simkit  and  event  graph  modeling  is  summarize 
in  table  1 .  More  information  on  how  to  use  Simkit  to  create  DBS  models  can  be  found  in 
(Buss,  2002).  Simkit  can  be  downloaded  from  the  internet  at  the  following  URL; 
http://diana.or.nps.naw.mil/simkit/ 
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Event  Graph 

Simkit 

Parameter 

Private  instance  variable,  setter  and  getter 

State  Variable 

Protected  instance  variable  getter,  no  setter 

Event 

‘do’  method 

Scheduling  Event 

Call  to  waitDelayO  in  scheduling  event’s  ‘do’  method 

Run  Event 

Reset()  method  to  initialize  state  variables; 

doRun()  method  to  fire  Property  Change  events  for  time- 
varying  state  variables 

Events  scheduled  from  Run  event 

Call  to  waitDelayO  doRun()  method 

Table  1 .  Relationship  between  Simkit  and  Event  Graph  modeling 


B,  DES  MODEL  FOR  TRADITIONAL  ENGINE  REPAIR  SYSTEM 

As  mentioned  in  ehapter  one,  this  thesis  builds  on  Sehoehs’  thesis  work.  His 
model  simulates  the  eurrent/traditional  F414-GE-400  engine  repair  proeess.  The  three 
levels  of  maintenanee,  0-level,  I-level  and  D-level  are  represented.  Simulates  F/A-E/F 
flight  sehedules  and  engine  failures  are  used  to  populate  the  engine  proeess.  The  twelve 
Java  elasses  of  Sehoeh’s  DES  model  follow  with  a  brief  deseription  (Sehoeh,  2003). 

1.  Module  Type  Class 

This  elass  sets  up  the  types  of  modules  that  ean  be  used  in  a  simulation.  For  the 
F414-GE-400  engine  repair  proeess  simulation  there  are  six  types  of  modules. 

2.  Module  Class 

An  objeet  of  this  elass  represents  a  module.  The  module  ean  be  of  any  type 
speeifled  by  the  module  type  elass,  for  example:  fan.  A  failure  time  is  randomly  seleeted 
from  a  data  set  eorresponding  to  the  module  type.  This  objeet  traeks  eomponent  hours 
used  and  knows  when  it  has  reaehed  high  time  or  time  of  failure. 

3.  Engine  Blueprint  Class 

This  elass  details  the  speeifieations  eaeh  engine  must  meet:  position  and  type  of 
eaeh  module,  build  window  and  high  time  sehedule.  0-Eevel  engine  removal. 
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installation  and  troubleshooting  times  are  set  in  this  elass.  Additionally,  I-Level  engine 
inspection  time  and  module  removal/installation  time  for  each  module  type  are  set  in  this 
class. 

4,  Engine  Class 

An  object  of  this  class  represents  a  F-414-GE-400  engine.  It  is  made  up  of  one  of 
each  type  of  module,  for  a  total  of  six  modules.  It  tracks  engine  hours  and  screens  each 
module  if  it  has  reached  high  time  or  time  of  failure.  The  time  to  high  time  corresponds 
to  the  module  time  to  high  time  with  lowest  value.  The  time  to  failure  corresponds  to  the 
module  failure  with  the  lowest  value.  Each  engine  object  must  conform  to  the 
specifications  of  the  Engine  Blueprint  class. 

5,  F18  Hornet  Class 

An  object  of  this  class  represents  an  E/A-18E/E  aircraft.  The  E/A-18E/E  is 
simulated  by  using  two  engine  objects  and  one  ElightSchedule  object.  The  aircraft  object 
tracks  its  own  hours.  Time  to  high  time  corresponds  to  the  engine  time  to  high  time  with 
the  lowest  value.  Time  to  failure  corresponds  to  the  engine  time  to  failure  with  the 
lowest  value.  When  an  aircraft  reaches  time  of  failure  of  high  time  it  is  no  longer  flyable. 

6,  Flight  Schedule  Class 

This  class  provides  a  selection  of  schedules  an  F/A-18E/F  Hornet  object.  The 
schedules  are  arrays.  Elements  indexed  by  odd  numbers  in  the  array  are  ground  times 
(time  between  flights).  The  even  indexed  elements  in  the  array  are  flight  times.  The  final 
odd  placed  element  in  the  array  is  a  zero.  This  element  directs  the  E/A-18E/E  Hornet  to 
begin  the  flight  schedule  again. 

7,  O-level  Class 

An  object  of  this  class  represents  a  squadron.  When  a  supported  aircraft  reaches 
high  time  or  has  an  engine  failure,  the  squadron  removes  the  engine.  The  squadron 
requests  a  replacement  engine  from  its  supporting  I-level.  Upon  arrival,  the  replacement 
engine  is  installed  in  the  aircraft  by  the  O-level  object. 

8,  I-level  Class 

An  object  of  this  class  represents  an  AIMD.  It  delivers  REI  engines  to  the  O- 
levels  it  supports.  A  linked  list  keeps  track  of  all  good  modules  assembled  on  engines 
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awaiting  part  and  in  inventory.  The  best  engines  are  built  by  screening  and  using 
modules  with  the  largest  amount  of  time  remaining  before  high  time  is  reached. 

9,  D-level  Class 

An  object  of  the  class  represents  a  depot.  Like  an  AIMD,  the  depot  has  linked 
lists  which  keep  a  track  of  all  its  modules.  A  module  received  from  the  AIMD  is  placed 
in  the  needs  repair  linked  list. 

10,  F18  Simulation  Manager  Class 

The  entire  infrastructure,  setup  in  the  FI 8  Simulation  Setup  class  is  created  here. 
It  determine  out  how  many  modules  and  engines  are  needed  based  on  the  number  of 
aircraft  and  inventories  at  all  I  and  D  levels.  This  class  keeps  a  record  of  every  object 
made  in  the  simulation  and  therefore  serves  as  the  report  generator  for  the  simulation. 

11,  F18  Simulation  Randomness  Class 

All  randomness  in  the  simulation  is  generated  in  this  class.  Module  times  between 
failures  are  randomly  selected,  with  replacement,  from  data  that  contains  actual  failure 
times. 

12,  F18  Simulation  Setup  Class 

This  class  controls  the  entire  simulation.  This  class  is  used  to  construct  the 
infrastructure  to  be  modeled  in  the  simulation.  The  number  of  aircraft,  squadrons, 
AIMDs,  and  I-level  and  D-level  inventories  are  set  up  in  this  class.  An  entity  knows 
what  it  supports  because,  using  Simkit,  it  is  registered  to  listen  to  it.  Figure  10  shows  the 
listening  relationship  between  the  object  of  each  type  class. 

0  Level  objects  listens  to  Hornets  ILevel  objects  listens  to  OLevels  DLevel  objects  listens  to  ILevel 

/  \  /  \  /  \ 


F18Hornet  objects  OLevel  objects  ILevel  Objects  DLevel  Objects 


\  /  \  /  \  / 

Hornet  object  listens  to  supporting  OLevel  OLevel  object  listens  to  supporting  ILevel  ILevel  object  listens  to  supporting  DLevel 


Figure  10.  Listening  Relationship  of  Simulated  Objects 
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C.  CHANGES  TO  SCHOCH’S  MODEL 

Two  logistics  systems  are  modeled;  traditional  and  ALS.  Schoch’s  simulation, 
with  some  adjustments,  is  used  to  model  the  traditional  system.  His  simulation  is  altered 
for  the  ALS  model.  Differenees  for  the  traditional  model  inelude; 

•  For  failure  times,  Sehoeh  randomly  selected  failure  times  from  data  sets. 
In  this  thesis,  a  probability  distribution  is  fit  to  eaeh  data  set.  These 
distributions  are  used  to  generate  the  failure  times. 

•  Sehoeh  modeled  the  depot  in  detail.  This  thesis  has  less  depot  detail.  In 
this  thesis,  the  depot  has  unlimited  resourees.  Depot  inventory  levels  and 
maintenanee  man  hours  are  not  eonsidered  in  this  thesis.  Impact  of  depot 
level  support  is  captured  by  assigning  various  values  to  the  shipping  time 
between  the  depot  and  AIMDs. 

•  Variables  are  added  to  the  Sehoeh  model  to  traek  squadron  and  AIMD 
man  hours,  number  of  requisitions  made,  number  of  requisition  fdled, 
number  of  requisitions  gone  off  base,  number  of  in-flight  failures,  and 
number  of  missions  eompleted. 

In  addition  to  the  differenees  above,  for  the  ALS  system  changes  to  the  Sehoeh 
model  inelude: 

•  No  high  times. 

•  Prognosties  (the  ability  to  deteet  pending  failures). 


D,  DES  MODEL  FOR  ALS  ENGINE  REPAIR  PROCESS 

The  model  in  this  thesis  simulates  the  F-414-GE-400  engine  repair  proeess  in 
aeeordanee  with  the  ALS.  The  0-level  and  I-level  are  modeled  with  detail.  The 
simulated  F/A-18E/F  aireraft  are  enhaneed  with  prognostics.  If  an  impending  failure  is 
deteeted,  the  aircraft  send  the  information  to  eaeh  of  the  three  levels  of  maintenanee 
thereby  initiating  aetion  at  eaeh  level. 

1.  Engine  Blueprint  Class 

As  in  the  traditional  model,  all  engines  in  the  ALS  must  eonform  to  the 
speeifications  detailed  in  the  engine  blueprint  elass.  Prognostie  aeeuracy  and  detection 
time  before  failure  (lead  time  -remaining  life  after  deteetion  before  the  failure  oeeurs)  are 
set  in  this  elass. 
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2.  F18  Hornet  Class 

The  aircraft  object  keeps  a  track  of  its  own  flight  hours.  When  the  prognostics 
detect  an  impending  failure,  the  0-level  and  I-level  are  instantly  notified.  Autonomously, 
a  replacement  engine  is  delivered  to  the  squadron.  The  aircraft  keeps  making  its 
scheduled  flights  until  one  of  the  following  occur:  the  prognostic  works  properly  and  a 
replacement  engine  arrives,  the  prognostics  works  properly  and  the  impending  failure  is 
predicted  to  occur  during  the  next  flight,  or  the  prognostics  do  not  work  and  an  in  flight 
failure  occurs.  The  engine  detection  time  check  method  screens  for  these  events.  The  first 
two  events  result  in  the  grounding  of  the  aircraft.  If  the  prognostics  do  not  work  and  a 
failure  occurs,  then  the  ALS  system  responds  in  the  same  way  as  the  traditional  (Schoch) 
system.  Additionally,  a  false  positive  time  is  generated  for  each  aircraft  object.  When  the 
aircraft  reaches  the  false  positive  time,  a  module  is  randomly  chosen  for  the  prognostics 
detect  an  impending  failure.  The  ALS  F-18  Hornet  event  graph  is  displayed  Appendix  B. 

3.  O-Level  Class 

The  0-level  class  keeps  track  of  engine  requests  made  from  all  the  aircraft  it 
supports.  Additionally,  it  tracks  maintenance  hours.  If  the  engine  to  be  replaced  has 
failed,  the  squadron  responds  in  the  traditional  manner  and  begins  troubleshooting.  If  the 
request  was  made  by  the  ALS,  the  squadron  does  not  troubleshoot  the  engine.  The 
squadron  waits  until  the  replacement  engine  arrives  then  replaces  the  engine.  The  ALS  O- 
level  event  graph  is  displayed  in  Appendix  C. 

4.  I-Level  Class 

The  I-level  class  keeps  track  of  engine  requests  made  from  all  the  squadrons  it 
supports.  Additionally,  it  tracks  the  number  of  RFI  modules  ordered  from  and  received 
from  the  depot.  It  also  tracks  its  maintenance  hours. 

Engines  received  from  the  squadron  have  either  failed  or  have  an  impending 
failure  that  has  been  detected.  If  an  engine  is  received  due  to  the  prognostics, 
troubleshooting  is  not  required.  If  a  failed  engine  is  received,  troubleshooting  is  required. 

Modules  that  have  failed  or  those  with  detected  impending  failures  are  removed 
from  the  engine  and  sent  to  the  depot  for  repair.  In  the  case  where  module  inventory  is 
empty,  additional  modules  may  be  removed  from  an  engine  in  order  to  complete  the 
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assembly  of  another  engine.  This  praetice  is  called  cannibalizing.  Cannibalizing  is  an 
allowed  practice  at  the  I-level.  The  ALS  I-level  Event  graph  is  displayed  in  Appendix  D. 

5,  D-level  Class 

The  depot  always  has  a  replacement  module  always  on  hand.  The  level  of  service 
of  the  depot  is  captured  by  altering  the  I-level  to  D-level  shipping  times.  The  D-Level 
class  keeps  a  track  of  the  number  of  requisitions  made  to  the  depot.  The  D-Level  Event 
graph  is  displayed  in  Appendix  E. 

6,  Stochastic  Class 

Eailures  depend  on  module  type.  The  probability  distributions  for  module  failures 
are  set  in  this  class.  Additionally,  the  distribution  that  generates  the  false  positive  signals 
is  set  here. 

E.  CONCLUSION 

DES  models  have  been  developed  to  simulate  the  traditional  logistic  system  and 
the  ALS.  The  characteristics  and  behavior  of  the  system  components  are  captured  in  the 
DES  models.  Java  and  Simkit  are  efficient  tools  in  producing  versatile  models. 
Parameters  can  be  changed  easily  to  evaluate  the  systems.  Modification  can  be  made  to 
the  DES  models  to  simulation  other  weapons  platforms,  specifically  the  JSE.  However, 
before  using  the  models,  their  validity  needs  to  be  determined.  As  a  partial  effort  towards 
this,  a  stochastic  model  was  formulated  and  compared  with  the  simulation.  This 
stochastic  model  is  described  in  the  following  chapter. 
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IV.  STOCHASTIC  MODELS 


Analytical  models  produce  closed  form  expressions  for  MOEs.  The  use  of  an 
analytieal  model  in  the  thesis  is  for  comparison  to  the  DES  models.  The  results  from  the 
stochastic  models  are  used  to  partially  validate  the  DES  models.  The  traditional  and  AES 
systems  ean  be  modeled  using  stoehastic  models.  The  intent  is  not  to  assess  the  entire 
logistie  proeesses  using  stoehastie  models.  Aireraft  engine  time  between  failures  and 
repair  times  are  used  to  approximate  the  long  run  average  rate  of  failure  and  the 
operational  availability  of  the  engine.  The  long  run  average  rate  of  failure  is  the  number 
of failures  per  flight  hour  (EPE).  The  operational  availability  is  the  long  run  proportion  of 
time  the  aireraft  is  up.  The  I  level  and  the  D  level  are  not  eaptured  in  the  stochastic 
models.  The  output  of  the  DES  model  and  the  stoehastie  model  results  are  eompared 
under  the  same  model  assumptions. 


A,  ASSUMPTIONS  AND  DES  MODEL 

Eaeh  modular  lifetime  distribution  is  assumed  to  be  exponentially  distributed  with 
a  mean  of  1,000  hrs.  Repair  time  of  an  aircraft  is  constant.  If  a  failure  oecurs,  repair  time 
is  three  hours.  If  prognostic  detects  the  impending  failure,  repair  time  is  two  hours.  The 
additional  hour  of  repair  is  added  for  a  failure  beeause  trouble  shooting  is  required  to 
isolate  the  problem.  A  repair  returns  the  aireraft  to  as  good  as  new. 

The  parameters  listed  above  are  used  in  the  DES  models  are  set  to  the  above 
values.  The  stoehastie  models  assume  a  system  (aireraft)  has  two  modes:  in-use  or  not- 
in-use.  System  modes  are  set  to  in-use  until  a  failure  oeeurs.  During  repairs,  systems  are 
in-use.  After  the  repairs  are  eomplete,  systems  return  to  the  in-use  mode.  The  DES 
models  are  modified  to  represent  this  assumption  for  eomparison  with  the  stoehastie 
analytieal  models.  Eor  comparison  runs  only,  aircraft  are  seheduled  for  a  single  long 
flight.  A  two-year  flight  duration  is  used  as  an  arbitrarily  large  value.  The  flight  is  only 
interrupted  if  a  failure  oeeurs  and  or  an  impending  failure  is  detected.  Immediately  upon 
repair  the  aircraft  continues  its  mission.  Eaeh  simulation  run  is  replicated  100  times. 
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Each  DES  model  result  is  the  mean  of  the  100  replications.  A  discussion  of  sample  size 
selection  is  included  in  section  D  of  Chapter  V. 


B,  RENEWAL  PROCESS 

1,  Module  Failures  Process  as  a  Renewal  Process 

Eirst,  a  renewal  proeess  is  defined,  and  then  counting  the  number  of  module 
failures  is  shown  to  satisfy  the  definition. 

If  the  sequence  of  times  between  events  {A^Aj...}  is  independent  and 
identically  distributed,  then  the  eounting  process  t  >  0  }  is  said  to 

be  a  renewal  proeess  (Ross,  2003). 

The  nonnegative  variables  denote  the  time  between  the  (n-l)st  and  nth  events.  An 
event  is  a  renewal.  Consider  a  module,  time  between  module  failures  is  nonnegative  and 
independent  and  identically  distributed.  Therefore,  counting  the  times  a  module  fails  is  a 
renewal  proeess.  In  addition,  allowing  the  module  failures  to  be  exponentially  distributed 
makes  the  proeess  a  Poisson  proeess.  The  Poisson  proeess  has  unique  properties  which 

make  formulation  easier.  The  rate  of  a  Poisson  proeess  is  /I  =  — ^ — . 

mean 

2,  Aircraft  Failures  as  a  Poisson  Process 

An  F-18/A  aircraft  has  two  engines.  Eaeh  engine  has  twelve  modules.  The 
aircraft  engine  failure  process  is  the  sum  of  the  twelve  independent  module  failure 
Poisson  processes.  The  sum  of  independent  Poisson  processes  is  a  Poisson  process  with 
a  rate  equal  to  the  sum  of  the  rates  of  the  independent  processes  (Ross,  2003). 

12 

As  a  result: /I  = /I,  +2,  +..  +  2,,  = - =  0.012  .  Where:  2  is  the  rate  of  failure  of  an 

1000 

aircraft,  2  is  the  rate  of  the  nth  module  and  2,  =2,  = ..  =  2,,  =  . 

1000 

3,  Long  Run  Average  Failure  Rate 

The  average  rate  of  a  renewal  process  converges  to  the  reciprocal  of  the  expected 
value  of  time  between  two  consecutive  events  as  time  approaches  infinity. 

^ —  as  t  — >  00  (Ross,  2003) 
t  E[A] 
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For  the  Poisson  process,  inter  arrival  times  are  exponentially  distributed.  The 
expected  value  of  the  exponential  process  is  the  reciprocal  of  its  rate  A  .  Therefore,  the 
long  average  rate  of  a  Poisson  process  is  \l{\l  A)  =  A . 

4.  The  Aircraft  Repair  Process  as  a  Renewal  Process 

Formulation  of  the  failures  is  already  developed  in  part  three  of  this  section. 
However  the  aircraft  is  also  subject  to  repair.  Repair  times  are  assumed  constant  and  are 
independent  and  identically  distributed.  Let  R(t)  be  the  number  of  repair  completions 
and  let  c  be  the  repair  time.  is  a  renewal  process.  For  the  model  the  expected  time 
between  repair  completions  is  A  +  c  . 

5.  The  Renewal  Reward  Process  and  Long  Run  Average  Availahility 

For  a  renewal  process,  suppose  at  each  interarrival  time  X  a  reward  is  received. 
Let  A  be  the  reward  received  at  each  renewal.  Let  Z(t)  represent  the  total  reward 
earned  by  time  t .  Z{t)  is  called  a  renewal  reward  process. 

The  lim  (Ross,  2003).  In  other  words,  the  long  run  average  rate  of 

t  E{X^ 

renewal  reward  process  is  the  expected  amount  of  reward  per  cycle  divided  by  the  length 
of  the  cycle.  As  a  result  the  long  run  average  rate  of  availability  is  the  expected  time  a 
device  is  up  in  a  cycle  divided  by  the  expected  length  of  the  cycle  (Jacobs,  2006).  For  our 

A, 

model  this  translates  to:  long  run  average  rate  of  availability  is -  . 

X  +  c 


C.  VALIDATING  THE  TRADITIONAL  SYSTEM 

The  traditional  system  enforces  an  age  replacement  policy.  Modules  are  replaced 
upon  failure  or  upon  reaching  a  predetermined  age. 

1,  Age  Replacement  Policy  and  Long  Run  Average  Failure  Rate 

For  a  Poisson  process,  the  long  run  average  failure  rate  for  the  age  replacement 
policy  is  the  same  as  the  long  run  average  failure  rate  as  replacing  components  only  upon 
failure.  A  proof  of  this  statement  is  provided  in  Appendix  F. 

12 

The  long  average  rate  of  failure  is:  X  =  ^  0.012/  flighthour  . 
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The  traditional  DBS  model  yields;  0.01094  standarderror  =  0.0003 . 


The  relative  differenee  is: 


|0.012-0.01094| 

0.012 


8.8%. 


2.  Long  Run  Portion  of  Time  an  Aircraft  is  Up 

The  expeeted  time  between  failures  is  —  =  .  The  expeeted  repair  time  is  three 

X  12 

hours.  The  expeeted  time  between  repair  eompletions  is:  +  3  . 

The  long  run  availability  rate  is:  +  3)  =  0.9653  . 

The  traditional  DBS  model  yields;  0.9682  standarderror  =  .001. 

.  10.9653 -0.9682| 

The  relative  ditterenee  is  ^ =  0.3%. 

0.9653 


D,  VALIDATING  THE  ALS 

Up  to  this  point,  the  traditional  system  has  been  diseussed.  Changes  to  the 
traditional  stoehastie  model  are  made  to  aeeount  for  the  deteetion  eapabilities  of  the  ALS. 
If  prognosties  work  the  impending  failure  is  deteeted  and  the  module  is  replaeed  before 
failure.  If  the  prognosties  do  not  work  the  module  fails. 

The  eounting  of  module  failures  oeeur  is  a  Poisson  proeess  with  rate  X  .  The  ALS 
model  is  as  follows.  Baeh  failure  is  assoeiated  with  an  independent  trial.  If  the 
prognosties  work  the  trial  is  a  sueeess.  Let  p  be  the  prognostie  aeeuraey,  then  the 
probability  of  sueeess  is  p .  The  probability  of  a  failure,  the  prognosties  did  not  work, 
isl-/?.  Let  L(t)be  the  number  of  failures,  then  L(t)is  a  Poisson  process  with 
rate Xx{l- p)  (Jacobs,  2006). 

1.  Long  Run  Average  Failure  Rate 

Let  o  =  0.9 ,  then  the  long  run  average  failure  rate  is  =  0.0012  . 

The  ALS  model  yields;  0.00138  standarderror  =  .00196 . 
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.  0.0012-0.00138 

The  relative  diiierenee  is:  - =  15%. 

0.0012 

2,  Long  Run  Average  Availability  Rate 

The  expeeted  time  between  failures  is  —  =  . 

The  expeeted  repair  time  is  (3  x  (i  -  p))  +  (2  x ;?) . 


The  expeeted  time  between  repair  eompletions  is  +  (3  x  (i  -  p))  +  (2  x  p). 
Let  p  =  0.9 ,  then  the  long  run  average  failure  rate  is: 


(^)  /(^  +  (3  X  (1  -  p))  +  (2  X  p))  =  0.997486  . 


The  ALS  DBS  model  yields:  0.9956  standarderror  =  .00057 . 


.  0.997486-0.9956 

The  relative  ditterenee  is:  - =  0.18%. 

0.997486 


E.  CONCLUSIONS 

The  stoehastie  models  are  simple  and  ean  be  used  to  provide  a  solution  quiekly. 
However,  stoehastie  models  are  low  resolution.  Stoehastie  models  eapture  the  general 
overall  behavior  of  the  logistie  system  and  are  used  to  eheeked  higher  resolution 
simulation  models.  The  simulation  models  pass  faee  validation.  Although  this  does  not 
eonstitute  a  eomplete  validation  of  the  DBS  models,  matehing  the  analytieal  results  gives 
eonfidenee  to  the  DBS  models’  correetness.  The  DBS  models  are  required  to  study  the 
entire  engine  proeess  in  detail.  The  next  step  is  to  simulate  the  traditional  and  logistie 
system  using  the  DBS  models. 
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V.  RESULTS  AND  DISCUSSION 


This  chapter  discusses  data  generation,  MOEs  and  design  faetors.  A  Latin 
Hypercube  design  is  used  to  derive  maximal  information  in  a  reasonable  number  of  runs. 
Simulation  input  and  output  for  both  the  traditional  system  and  the  ALS  is  also 
addressed. 

A,  DESIGN  OF  SIMULATION  EXPERIMENTS 

1.  Measures  of  Effectiveness 

Within  the  DOD,  the  most  common  MOL  for  aircraft  readiness  is  operational 
availability.  Operational  availability  is  analyzed  as  a  response  variable.  However, 
operational  availability  can  be  misleading.  Operational  availability  does  not  indicate  how 
often  aircraft  are  used  or  how  often  they  fail.  If  a  squadron  with  flyable  aircraft  did  not 
fly  for  a  year,  the  operation  availability  for  the  year  is  100%.  In  this  instance,  no 
conelusions  ean  be  made  about  the  logistic  system  in  plaee.  Other  indictors  also  should 
to  be  considered  in  order  to  get  a  more  aeeurate  picture  of  the  logistie  system. 

Other  MOEs  or  response  variables  considered  are:  MMHE  and  EPM.  EPM  is  not 
the  same  as  EPH,  whieh  was  used  in  the  previous  chapter.  Both  of  these  response 
variables  are  directly  proportional  to  cost  and  provide  insight  to  the  effectiveness  of 
allocating  parts  and  personnel. 

2.  Design  Factors 

Design  factors  are  the  independent  variables  considered  as  predictors  for  the 
MOEs.  The  traditional  system  has  four  design  factors:  module  reliability,  build  window, 
depot-turn-around-time  and  inventory  levels.  The  ALS  system  has  six  design  factors: 
module  reliability,  detection  lead  time,  prognostic  accuracy,  false  positive  rates,  depot 
turn  around  times  and  inventory  levels.  The  design  factors  are  analyzed  to  determine 
their  influence  on  each  of  the  MOEs. 

The  baselines  for  both  logistic  systems  used  to  compare  all  results  are  listed  in 
Table  6  of  Appendix  A.  The  design  faetors  and  the  values  they  ean  take  are  summarized 
in  Table  2  below.  Module  reliability  is  increased  by  increasing  the  mean  of  each 
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distribution.  Inventory  levels  for  each  module  are  simultaneously  increased  with  values 
from  the  table.  Detection  lead  time  is  the  amount  of  life  remaining  on  a  module  once  an 
impending  failure  is  detected.  False  positive  rates  are  exponentially  distributed  with  the 
mean  values  indicated  in  Table  2. 


Design  Factor 

Values 

Module  reliability  (hrs) 

0,100,200,300,400,500,600,00,800,900,1000 

Build  Window  (hrs) 

50,  100,  150,  200,250,300,350,400,450,500 

Depot  Turn  Around  (days) 

5,10,15,20,25,30,35,40 

Detection  Lead  Time  (hrs) 

15,20,25,30,35,40 

Prognostic  Accuracy 

0.90,0.91,0.92,0.93,0.94,0.95,0.96,0.97,0.98,0.99,1.0 

False  Positive  Means  (hrs) 

450,  500,550,600,650,700 

Inventory  (modules) 

0,1,2,3,4,5,6,7,8,9,10 

Table  2.  Design  Factors  and  Ranges 


A  common  technique  in  experiment  design  is  to  run  the  model  using  a  baseline, 
additional  runs  are  made  changing  one  design  factor/predictor  variable  at  a  time. 
Predictor  variables  with  the  greatest  impact  are  then  selected  as  significant  factors.  This 
approach  is  incomplete  because:  significant  combinations  of  design  factors  may  be 
overlooked.  For  the  full  factorial  design,  adding  more  design  factor  levels  results  in 
numerous  required  simulation  runs.  Fractional  factorial  designs  reduce  the  number  of 
runs,  but  introduce  confounding  of  interactions  between  design  factors.  Key  interactions 
may  be  masked  or  confounded  by  the  main  effects  (first  order  effects). 

3.  Latin  Hypercubes 

A  full  factorial  design  considers  factor  interactions.  Such  a  design  for  all 
combinations  of  the  four  predictor  variables  for  the  traditional  model  would  require  9,680 
design  points.  A  full  factorial  design  of  all  combinations  of  the  six  predictor  variables  for 
the  ALS  model  would  require  383,328  design  points.  Fortunately,  other  methods  can  be 
used  to  construct  valid  confidence  estimates  for  the  MOEs  with  fewer  design  points. 

Latin  Hypercubes  are  a  very  good  all-purpose  design,  particularly  when  factors 
are  quantitative  because  of:  efficiency,  space-filling,  design  flexibility,  and  analysis 
flexibility  (Sanchez,  2006).  The  nearly-orthogonal  Latin  Hypercube  (NOLH)  design  is 
used  for  this  experiment.  A  spreadsheet  written  by  Professor  Susan  Sanchez,  Naval 
Postgraduate  School,  was  used  to  generate  seventeen  design  points  for  each  system. 
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B,  SIMULATION  INPUT  AND  OUPUT 

The  model  input  and  outputs  are  summarized  in  Table  3  and  Table  4.  Table  3 
refers  to  the  traditional  model.  Table  4  refers  to  the  ALS  model.  The  inputs  are  the 
design  faetors.  The  outputs  are  the  MOEs.  Each  row  is  a  design  point  (simulation  run) 
with  its  corresponding  output.  Each  simulation  run  is  replicated  100  times.  The 
corresponding  output  is  the  mean  of  the  100  replications.  The  following  section  is  a 
discussion  of  the  sample  size  detection. 


Traditional  Model 


Input 

Output 

Reliability 

Depot 

Inventory 

Build 

Availability 

FPM 

MMHF 

(hrs) 

(days) 

(modules) 

(hrs) 

400 

960 

8 

200 

0.9962 

0.0176 

0.3702 

200 

360 

9 

300 

0.996 

0.0207 

0.3956 

200 

480 

1 

150 

0.9956 

0.0206 

0.4202 

300 

600 

3 

500 

0.9965 

0.0191 

0.3664 

800 

960 

4 

100 

0.9955 

0.0133 

0.3504 

1000 

360 

4 

400 

0.9964 

0.0123 

0.2826 

240 

10 

200 

0.9967 

0.0143 

0.3288 

600 

840 

8 

0.997 

0.0155 

0.3163 

500 

5 

0.9966 

0.0165 

0.3405 

2 

350 

0.997 

0.0145 

0.3097 

900 

720 

1 

250 

0.9953 

0.0128 

0.3032 

900 

600 

9 

400 

0.9965 

0.013 

0.2888 

800 

480 

7 

500 

0.9971 

0.0138 

0.2936 

300 

120 

6 

450 

0.9964 

0.0192 

0.3676 

100 

720 

6 

150 

0.9954 

0.0223 

0.443 

400 

840 

0 

350 

0.9965 

0.0178 

0.3577 

500 

240 

3 

100 

0.996 

0.0162 

0.3818 

Depot  =  depot  turn  around  time  (days) 

Table  3.  Traditional  System  Input  and  Output 
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ALS 


Ini 

put 

Output 

Reliability 

Depot 

Inventory 

Prognostics 

False 

Lead 

Availability 

FPM 

MMHF 

(hrs) 

(days) 

(module) 

(fraction) 

(hrs) 

(hrs) 

300 

40 

8 

0.94 

500 

40 

0.997 

0.0011 

0.266 

100 

15 

9 

0.96 

450 

25 

0.9966 

0.0009 

0.306 

100 

20 

1 

0.93 

600 

35 

0.9966 

0.0017 

0.3108 

200 

25 

3 

1 

600 

20 

0.9969 

0 

0.2753 

800 

40 

4 

0.91 

550 

15 

0.9978 

0.0012 

0.1976 

1000 

15 

4 

35 

0.998 

0.0002 

0.177 

600 

25 

0.9975 

0.0011 

0.2183 

600 

35 

8 

0.99 

650 

30 

0.9976 

0.0002 

0.2136 

500 

25 

5 

30 

0.9974 

0.0008 

0.2296 

5 

2 

15 

0.9977 

0.0005 

0.2044 

900 

30 

1 

0.94 

700 

30 

0.9979 

0.0007 

0.1867 

900 

25 

9 

0.98 

550 

20 

0.9979 

0.0002 

0.1853 

800 

20 

7 

0.9 

550 

35 

0.9978 

0.0013 

0.1995 

300 

5 

6 

0.99 

600 

40 

0.9971 

0.0002 

0.2587 

0 

30 

6 

0.92 

700 

20 

0.9995 

0.0016 

0.3057 

400 

35 

0 

0.97 

500 

30 

0.9972 

0.0005 

0.2439 

400 

10 

3 

0.91 

500 

25 

0.9973 

0.0016 

0.2481 

F=  False  positive  rate 

Depot  =  Depot  turn  around  time 

Lead  =  Detection  Lead  Time 

Table  4.  ALS  Model  Input  and  Output 

C.  POWER  ANALYSIS  AND  SAMPLE  SIZE  SELECTION 

Power  analysis  is  used  to  determine  sample  size  (number  of  replieations  required) 
based  on  a  (probability  of  a  type  II  error)  of  O.I.  Power  is  1-  [5  whieh  is  0.9.  The 
(probability  of  a  type  I  error),  a  ,  is  set  to  0.05.  The  model  being  evaluated  is:  Response 
variable  =  (true  mean)  +  (faetor  effeet)  +  (error).  If  the  null  hypothesis  is  true,  then  the 
faetor  effeet  is  zero.  The  sample  size  required  to  deteet  a  deviation  in  the  true  mean  due 
to  a  non-zero  faetor  effeet  is  determined.  To  deteet  a  smaller  effeet  a  larger  sample  size 
is  required. 

IMP  statistieal  software  is  used  to  ealeulate  the  power  eurve.  An  estimate  of 
standard  deviation  is  needed  to  perform  the  ealeulation.  Standard  deviation  is  estimated 
by  repeating  eaeh  design  point  100  times  and  ealeulating  sigma-hat.  Figure  11  is 
generated  to  determine  the  sample  size  for  operational  availability  in  the  traditional 
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logistic  model.  The  graph  is  a  0.9  power  eurve.  For  deviations  of  0.005,  45  repiieations 
are  needed  to  detect  that  deviation  with  a  probability  of  0.90. 


I  Sample  Size 

One  Mean 

Error  Std  Dev  Power  Alpha 


Figure  11.  0.9  Power  Curve 

A  0.90  power  curve  is  eomputed  for  eaeh  MOE  in  eaeh  model  by  using  the  same 
proeedure.  The  results  are  summarized  in  table  5.  All  effeets  deteeted  are  smaller  than 
0.008.  The  largest  n  required  is:  70.  With  100  replications  smaller  deviations  ean  be 
deteeted  with  a  power  of  0.90. 


Traditional  Logistic  System 

ALS 

sample  size 

deviation 

sample  size 

deviation 

Operational 

45 

0.005 

60 

0.005 

FPF 

45 

0.0015 

70 

0.0075 

FPM 

45 

0.0020 

60 

0.005 

MMHF 

60 

0.0020 

60 

0.006 

Table  5.  Power  Analysis  results. 
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D,  ANALYSIS  OF  THE  SIMULATION  OUTPUT 

Each  simulation  produces  three  output  MOEs.  Analysis  of  the  MOEs  for  both  the 
AES  and  traditional  logistie  system  served  to  eompare  both  models. 

The  intent  of  the  simulation  is  to  study  the  effeets  of  the  design  faetors  on  eaeh  of 
the  MOEs.  To  gain  insight  into  the  relationship  of  the  variables,  linear  regression  models 
are  fitted  to  the  generated  data.  The  regression  models  are  a  result  of  using  stepwise 
regression  and  residual  analysis.  The  stepwise  algorithm  adds  and  drops  terms  by  using 
the  Akaike’s  Information  Criterion.  The  P-value  is  the  probability  the  effeet  of  a  variable 
equals  zero.  Variable  with  P-value  greater  then  0.001  are  not  eonsidered.  The  smaller  the 
P-value  the  more  evidenee  there  is  against  the  null  hypothesis;  the  predietor  variable  has 
no  effeet.  If  diagnostie  plots  do  not  support  the  modeling  assumptions  of  normal  errors 
and  homoseedastieity,  mathematieal  transformations  are  applied  until  the  assumptions  are 
satisfied.  Transformations  inelude:  logarithmie,  reeiproeal,  aresine  and  square  root. 
Additionally,  if  the  relationship  between  the  response  variable  and  the  predietor  variables 
is  not  linear,  polynomials  of  degree  two  are  used  for  the  regression. 

IMP  statistieal  software  is  used  to  perform  multiple  regression  and  to  generate 
graphs  to  summarize  the  analysis.  Eor  eaeh  MOE,  linear  regression  models  are  produeed 
to  deseribe  its  relationship  with  the  input  variables  of  eaeh  system.  Seatter  plot  matriees 
are  used  to  eheek  the  pairwise  relationship  between  all  variables.  When  two  or  more 
predietor  variables  are  found  to  be  signifieant  a  eorrelation  matrix  is  generated  to  verify 
they  are  independent.  Contour  plots  are  used  to  eompare  the  value  of  predietor  variable 
pairs  for  an  MOE.  The  eontour  values  are  plotted  from  the  results  of  the  17  design 
points.  The  eontour  plot  is  not  a  feasible  region  and  it  may  or  may  not  eontain  the 
optimum  point. 

The  predietive  eapability  of  eaeh  model  is  verified  by  exploring  inputs  not  already 
used.  A  speeifie  eombination  of  design  faetors  yields  the  optimal  value  for  eaeh  MOE. 
An  optimal  solution  is  sought  for  eaeh  MOE:  maximum  operational  availability, 
minimum  EPM  and  maximum  MMHE.  The  optimal  solution  is  found  for  eaeh  linear 
model.  The  DES  model  is  then  run  with  the  optimal  solution.  The  output  generated  by 
inputs  for  the  DES  model  and  linear  model  are  eompared.  Outputs  close  in  value  indieate 
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an  adequate  linear  model.  It  is  important  to  note  the  linear  models  are  valid  for  speoifie 
predietor  variable  ranges.  Values  outside  the  define  ranges  from  table  2  are  not  explored. 

A  linear  model  with  predietive  eapability  ean  provide  as  mueh  insight  as  the  more 
eomputationally  intensive  DBS  model.  Additionally,  a  linear  model  ean  be  put  in  an 
Exeel  spreadsheet  whieh  is  easier  and  quieker  to  use.  For  FPM  and  MMHF,  linear  models 
are  be  used  to  eompare  the  traditional  system  and  the  AFS. 

1.  Traditional  Repair  System 

Data  generated  by  the  DES  traditional  repair  system  model  resulted  in  good 
predietive  models  for  FPM  and  MHPF,  but  not  for  operational  availability. 

a.  Operational  Availability 

Regression  is  used  to  fit  a  linear  model  to  explain  the  relationship  between 
the  predietor  variables:  module  reliability,  build  window,  depot  turn  around  time,  and 
inventory,  and  the  response  variable  operational  availability.  Figure  12  is  the  IMP 
regression  report.  The  parameter  estimates  table  of  the  report  gives  the  estimated 
eoeffieients,  their  standard  errors  and  the  corresponding  t-statistics  to  test  the  partial 
effect  of  each  variable.  The  for  this  least  square  fit  is  0.503,  indicating  the  regression 
model  does  not  adequately  explain  the  variation  in  operation  availability.  Build  windows 
is  the  only  predictor  variable  kept  in  the  model.  Inventory  and  depot  turn  around  time 
were  anticipated  to  be  significant  in  terms  of  operational  availability.  Not  being  selected 
for  the  model  does  not  mean  they  are  not  important.  For  the  predictor  variable  ranges 
from  table  3,  inventory  and  depot  turn  around  time  do  not  limit  operational  availability. 
For  the  traditional  logistic  system,  I-level  inventory  equivalent  to  that  from  table  6  with  a 
guaranteed  depot  turn  around  time  of  40  days  is  all  that  is  required.  In  other  words,  once 
at  this  level  increasing  the  I-level  inventory  or  decreasing  the  depot  turn  around  time  has 
a  minimal  effect  on  operational  availability.  At  this  level,  the  main  driver  for  operational 
availability  is  build  windows. 
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Response  Operational  Availability 


RSquare  0.503204 

RSquare  Adj  0.470084 

Root  Mean  Square  Error  0.000412 
Mean  of  Response  0.996276 

Observ  ations  (or  Sum  Wgts)  17 

Analysis  ofVariance 


Source  1 

DF 

Sum  of  Squares  Mean  Square 

F  Ratio 

Model 

1 

0.00000258  0.0000026 

15.1935 

Error 

15 

0.00000255  1. 

6992e-7 

Prob  >  F 

C.  Total 

16 

0.00000513 

0.0014 

Lack  Of  Fit  I 

Source 

DF  Sum  of  Squares  Mean  Square  F  Ratio 

Lack  Of  Fit 

7  0.00000161 

2.2984e-7  1.9561 

Pure  Error 

8  0.00000094 

1.1 75e-7  Prob  >  F 

Total  Error 

15  0.00000255 

0.1836 

Max  RSq 

0.8168 

Parameter  Estimates  | 

Term 

Estimate  Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

0.9953838  0.00025 

3983.5 

<  0001 

buildwindows 

0.0000029  7.559e-7 

3.90 

0.0014 

Effect  Tests 


Source  Nparm  DF  Sum  of  Squares  F  Ratic  Prob  >  F 

buildwindows  1  1  0.00000258  15.1935  0.0014 


Figure  12.  JMP  Regression  Report  for  Traditional  System  Operational  Availability 
b.  Failures  Per  Flight  Mission 

Regression  is  used  to  fit  a  linear  model  to  explain  the  relationship  between 
the  predictor  variables:  module  reliability,  build  window,  depot  turn  around  time  and 
inventory,  and  the  response  variable  FPM.  Figure  13  is  the  JMP  regression  report,  i?"  for 
this  least  square  fit  is  0.998,  indicating  the  regression  does  a  good  job  in  accounting  for 
the  variability  in  FPM.  The  parameter  estimates  table  of  the  report  gives  the  estimated 

coefficients,  their  standard  errors  and  the  corresponding  t-statistics  to  test  the  partial 

44 


effect  of  each  variable.  Module  reliability  is  the  only  predictor  variable  selected  for  the 
linear  model.  Build  windows  is  the  next  significant  variable,  but  not  significant  enough 
to  be  included  in  the  model. 


R  Square 

0.998113 

RSquare  Adj 

0.997844 

Root  Mean  Square  Error 

0.000145 

Mean  of  Response 

0.016441 

Observations  (or Sum  Wgts) 

17 

Analysis  of  Variance 

F  Ratio 
3703.262 
Prob  >  F 
<.0001 

[  Lack  Of  Fit 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Lack  Of  Fit 

7 

5.43588e-8 

7.7655e-9 

0.2265 

Pure  Error 

7 

0.00000024 

3.4286e-8 

Prob  >  F 

Total  Error 

14 

2.94359e-7 

0.9657 

Max  RSq 

0.9985 

[  Parameter  Estimates 


Source  DF  Sum  of  Squares  Mean  Square 

Model  2  0.00015573  0.000078 

Error  14  0.00000029  2.103e-8 

C.  Total  16  0.00015602 


Term  Estimate  Std  Error  t  Ratio  Prob>|t 

Intercept  0.0220116  0.000088  250.42  <.0001 

Module  Reliability  (Flight  hrs)  -0.000011  1.302e-7  -85.51  <.0001 

(Module  Reliability  (Flight  hrs)-547.059)*(Module  Reliability  (Flight  hrs)-547.059)7. 1084e-9  5.52e-10  12.87  <.0001 

[Effect  Tests 


Source  N  parm 

Module  Reliability  (Flight  hrs)  1 

Module  Reliability  (Flight  hrs)*Module  Reliability  (Flight  hrs)  1 


Figure  13.  JMP  Regression  Report 


DF  Sum  of  Squares  F  Ratic  Prob  >  F 
1  0.00015375  7312.29  <.0001 

1  0.00000348  165.5464  <.0001 


for  Traditional  System  FPM 
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The  scatter  plot  matrix,  Figure  14,  indicates  there  is  no  collinearity 
between  module  reliability  and  build  windows.  The  plot  shows  a  linear  relationship 
between  module  reliability  and  FPM.  This  supports  the  fitted  model; 

0.022  -  (0.00001  *  ModuleReliability)  +  (0.00000007  *  (ModuleReliabilty  -  547.06)''2) 


Multivariate 
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Figure  14.  Scatter  Plot  Matrix  Traditional  System  FPM 
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Figure  15  is  a  contour  plot,  comparing  module  reliability  and  build 
windows  in  terms  of  FPM.  The  vertical  color  shading  pattern  suggests  failure  rate 
improves  with  the  increase  of  module  reliability  and  is  not  affected  by  build  windows. 
This  provides  further  support  for  the  fitted  model. 
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Figure  15.  Contour  Matrix  Traditional  System  FPM 
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Figure  16  is  a  plot  of  the  FPM  linear  model  of  the  traditional  logistic 
system  over  the  module  reliability  range.  The  fitted  regression  model  decreases  in  FPM 
when  module  reliability  increases.  Therefore,  the  largest  value  of  module  reliability  in  the 
model  range  produces  the  smallest  FPM.  The  highest  value  of  the  module  reliability  is 
1000  hrs.  Module  reliability  is  set  to  1000  hrs.  The  regression  model  produces  an  FPM 
of  0.01234.  With  module  reliability  fixed  at  1000  hrs,  the  DBS  model  for  the  traditional 
repair  system  produces  an  FPM  of  0.0125  with  standard  error  0.00004.  The  difference 
between  the  regression  model  and  the  simulation  model  is  1.28  %. 


Linear  Regression  Model  for  traditional 
system  FPM  over  increase  in  module 
reliability  range 


Module  Reliability 

Figure  16.  FPM  Linear  Regression  Model  for  Traditional  Repair  System 
c.  Maintenance  Man  Hours  Per  Flight  Hour 

Regression  is  used  to  fit  a  linear  model  to  explain  the  relationship  between 
the  predictor  variables:  module  reliability,  build  window,  depot  turn  around  time  and 
inventory,  and  the  response  variable  MMHF.  Figure  17  is  the  JMP  regression  report  that 
includes  a  correlation  matrix.,  i?"  is  0.964,  indicating  the  regression  does  a  good  job  in 
accounting  for  the  variability  in  MMFIF.  The  parameter  estimates  table  of  the  report  gives 
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the  estimated  eoefficients,  their  standard  errors  and  the  eorresponding  t-statistics  to  test 
the  partial  effect  of  each  variable.  Module  reliability  and  build  windows  are  the  predictor 
variables  selected  for  the  linear  model.  Module  reliability  and  build  windows  have 
approximately  equal  weight  on  the  expected  value  of  MMHF. 


RSquare  0.964953 

RSquare  Adj  0.959946 

Root  Mean  Square  Error  0.009308 
Mean  of  Response  0.348024 

Observ  ations  (or  Sum  Wgts)  17 

Analysis  of  Variance 

Source  DF  Sum  of  Squares  Mean  Square  F  Ratio 

Model  2  0.03339347  0.016697  192.7328 

Error  14  0.00121284  0.000087  Prob  >  F 

C.  Total  16  0.03460631  <.0001 

Param  eter  Estimates 


Term 


Estimate  Std  Error  t  Ratio  Prob>|t| 


Intercept  0.4632566  0.006733  68.81  <.0001 

Module  Reliability  (Flight  hrs)  -0.000141  0.000008  -16.59  <  0001 

buildwindows  -0.000126  0.000017  -7.24  <.0001 

Effect  Tests 


Source  Nparm  DF  Sum  of  Squares  F  Ratic  Prob  >  F 

Module  Reliability  (Flight  hrs)  1  1  0.02384449  275.2404  <  0001 

buildwindows  1  1  0.00454368  52.4484  <  0001 

Correlation  of  Estimates  | 

Corr 


InterceptModule  Reliability  (Flight  hrb^ildwindows 


Intercept  1.0000 

Module  Reliability  (Flight  hrs) -0.5456 
buildwindows  -0.6536 


-0.5456  -0.6536 

1.0000  -0.1853 

-0.1853  1.0000 


Figure  17.  Traditional  MMFIF  Regression  Model 
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The  scatter  plot  matrix,  Figure  18,  indicates  there  is  no  collinearity 
between  module  reliability  and  build  windows.  The  plot  shows  a  moderate  linear 
relationship  between  build  windows  and  MMHF.  However,  the  plot  shows  a  slightly 
stronger  linear  relationship  between  module  reliability  and  MMHF.  This  supports  the 
fitted  regression  model:0.46-(0.00014*reliabilty)-(0.000125*buildwindow) . 


200  400  600  8001000  100  200  300  400  500  600  .3  .35  .4  .4 


Figure  18.  Scatter  Plot  Matrix  Traditional  System  MMHF 
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Figure  19  is  a  contour  plot,  comparing  module  reliability  and  build 
windows  in  terms  of  MMFIF.  The  color  shading  forms  a  forty-five  degree  pattern 
suggesting  module  reliability  and  build  windows  are  nearly  equivalent  predictors  of 
MMFIF.  This  provides  further  evidence  in  support  of  the  fitted  regression  model. 
Additionally,  build  window  has  no  effect  on  MMHF  when  module  reliability  is  800  hrs 
or  greater. 
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Figure  19.  Contour  Plot  Traditional  System  MMFIF 


Figure  20  is  a  plot  of  the  MMHF  linear  model  of  the  traditional  repair 
system  over  the  build  window  range  with  no  improvement  in  module  reliability  range. 
Figure  21  is  a  plot  of  the  MMHF  linear  model  of  the  traditional  repair  system  over  the 
module  reliability  range  and  a  constant  build  window  of  500  hours.  The  fitted  regression 
model  predicts  a  decrease  in  MMHF  when  module  reliability  increases  or  when  build 
window  increases.  Linear  programming  is  used  to  optimize  the  MMHF  regression  model 
with  the  predictor  variable  ranges  as  constraints.  Setting  both  module  reliability  and 
build  window  to  the  highest  value  in  their  respective  range  yield  the  optimal  (smallest) 
MMHF.  Module  reliability  is  set  to  1000  hrs  and  build  window  is  set  500  hours.  The 
regression  model  produces  an  MMHF  of  0.2954.  The  DBS  model  for  the  traditional 
repair  system  produces  an  MMHF  of  0.2755  with  standard  error  0.000047.  The 
difference  between  the  regression  model  and  the  simulation  is  7.2%. 
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Linear  Regression  Model  for  traditional  system 
MMHF  over  build  window  range  and 
no  improvemnet  in  reliabilty 
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Figure  20.  MMHF  Regression  Model  for  Traditional  Repair  System 


Linear  Regression  model  for  traditional 
system  MMHF  over  increase  in  module  reliabilty  range 
with  constant  build  window  =500hrs 
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Figure  21 .  MMHF  Regression  Model  for  Traditional  Repair  System 
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2,  Analysis  of  ALS  Output 

Data  generated  by  the  DBS  ALS  model  resulted  in  good  predietive  models  for  the 
FPM  and  MMHF  but  not  for  operational  availability. 

a.  Operational  Availability 

Regression  is  used  to  fit  a  linear  model  to  explain  the  relationship  between 
the  predietor  variables:  module  reliability,  prognosties  aeeuraey,  false  positives,  depot 
turn  around  time,  inventory  and  deteetion  lead  time,  and  the  response  variable  operational 
availability.  Figure  22  is  the  JMP  regression  report.  The  parameter  estimates  table  of  the 
report  gives  the  estimated  eoeffieients,  their  standard  errors  and  the  eorresponding  t- 
statisties  to  test  the  partial  effeet  of  eaeh  variable.  The  value  is  0.14,  indieating  the 
linear  regression  model  does  not  explain  the  variation  in  operation  availability.  This 
suggests  the  seleeted  multiple  regression  model  prediets  poorly.  Module  reliability  and 
prognostie  aeeuraey  are  the  only  predietor  variables  kept  in  the  model.  Inventory  and 
depot  turn  around  time  were  antieipated  to  be  signifieant  in  terms  of  operational 
availability.  Not  being  seleeted  for  the  model  does  not  mean  they  are  not  important.  For 
the  speeifie  variable  ranges  from  table  4,  inventory  and  depot  turn  around  time  do  not 
limit  operational  availability.  For  the  ALS,  I-level  inventory  equivalent  to  that  from  table 
6  with  a  guaranteed  depot  turn  around  time  of  40  days  is  all  that  is  required.  In  other 
words,  at  these  speeifie  values  inereasing  the  I-level  inventory  or  deereasing  the  depot 
turn  around  time  has  a  minimal  effeet  on  operational  availability.  At  these  speeifie 
values,  the  main  driver  for  operational  availability  is  prognostie  aeeuraey  followed  by 
module  reliability. 
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Root  Mean  Square  Error 
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0.997518 

Observations  (or  Sum  Wgts) 

17 

Analysis  of  Variance 

Source  DF  Sum  of  Squares  Mean  Square  F  Ratio 

Model  2  0.00000104  5.2202e-7  1.1636 

Error  14  0.00000628  4.4862e-7  Prob  >  F 

C.  Total  16  0.00000732  0.3408 

Param eter  Estimates 

Term  Estimate  Std  Error  t  Ratio  Prob>|t| 

Intercept  1.0025573  0.005092  196.88  <.0001 

Module  Reliabilty  (flight  hrs)  5.9124e-7  5.347e-7  1.11  0.2874 

Prognostics  Accuracy  (%)  -0.005616  0.005347  -1.05  0.3113 

Effect  Tests 

Source  Nparm  DF  Sum  of  Squares  F  Ratic  Prob  >  F 

Module  Reliabilty  (flight  hrs)  1  1  5.48618e-7  1.2229  0.2874 

Prognostics  Accuracy  (%)  1  1  4.95023e-7  1.1034  0.3113 


Figure  22.  JMP  Regression  Output  for  ALS  Operational  Availability 
b.  Failures  Per  Mission 

Regression  is  used  to  fit  a  linear  model  to  explain  the  relationship  between 
the  predictor  variables:  module  reliability,  prognostics  accuracy,  false  positives,  depot 
turn  around  time,  inventory  and  detection  lead  time,  and  the  response  variable  FPM. 
Figure  23  is  the  JMP  regression  report  with  correlation  matrix  attached.  The  parameter 
estimates  table  of  the  report  gives  the  estimated  coefficients,  their  standard  errors  and  the 
corresponding  t-statistics  to  test  the  partial  effect  of  each  variable.  is  0.98,  indicating 
the  regression  does  a  good  job  in  accounting  for  the  variability  in  FPM.  Module 
reliability  and  prognostic  accuracy  are  the  two  predictor  variables  selected  for  the  linear 


54 


model.  Prognostic  accuracy  has  the  heaviest  weight  value  on  the  expected  value  of  FPM 
followed  by  module  reliability  and  the  interaction  between  prognostics  accuracy  and 
module  reliability. 


RSquare  0.981146 

RSquareAdj  0.976796 

Root  Mean  Square  Error  0.000085 

Mean  of  Response  0.000812 

Observ  ations  (or  Sum  Wgts)  17 

[  Analysis  of  Variance 

Source  DF  Sum  of  Squares  Mean  Square  F  Ratio 
Model  3  0.00000486  0.0000016  225.5073 

Error  13  9.34697e-8  7.19e-9  Prob  >  F 

C.  Total  16  0.00000496  <.0001 

I  Param  eter  Estimates 
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R 
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R  ^  module  reliability,  P=  prognostie  aeeuraey 


Figure  23.  JMP  Regression  Report  for  ALS  FPM 


55 


The  scatter  plot  matrix,  Figure  24,  indicates  there  is  no  collinearity 
between  module  reliability  and  prognostic  accuracy.  The  plot  shows  a  moderate  linear 
relationship  between  module  reliability  and  FPM.  However,  the  plot  also  shows  a 
stronger  linear  relationship  between  prognostic  accuracy  and  FPM.  This  supports  the 
fitted  regression  model: 

0. 167  -  (5.9e  -  7  *  reliability)  -  (0.016  *  prognostics)  +  (0.0001*  (reliability  -  505.9)  *  (prognostics  -  0.95) 


Multivariate 
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Figure  24.  Scatter  Plot  Matrix  ALS  FPM 
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Figure  25,  is  a  contour  plot,  comparing  module  reliability  and  prognostic 
accuracy  in  terms  of  FPM.  Note  the  color  pattern  tends  to  be  horizontal  and  slightly 
tilted,  indicating  prognostic  accuracy  is  more  important.  The  tilt  in  the  contour  plot 
indicates  including  an  interaction  between  module  reliability  and  prognostic  accuracy 
helps  predicting  FPM.  The  interaction  term  is  included  in  the  regression.  This  provides 
further  evidence  in  support  of  the  fitted  regression  model. 


Figure  25.  Contour  Plot  ALS  FPM 


Figure  26  is  a  plot  of  the  MMHF  linear  model  of  the  traditional  repair 
system  over  the  build  window  range  with  no  improvement  in  module  reliability  range. 
The  fitted  regression  model  predicts  a  decrease  in  FPM  when  prognostic  accuracy 
increases  or  when  module  reliability  increases.  Linear  programming  is  used  to  optimize 
the  FPM  regression  model  with  the  predictor  variable  ranges  as  constraints.  Setting  both 
prognostic  accuracy  module  reliability  to  the  highest  value  in  their  respective  range  yield 
the  optimal  (smallest)  FPM.  Prognostic  accuracy  is  set  to  1.0  and  module  reliability  is  set 
1000  hrs.  The  regression  model  produces  an  FPM  of  0.00005.  The  DBS  model  for  the 
ALS  produces  an  FPM  of  0.0  with  standard  error  0.0.  The  relative  difference  between  the 
regression  model  and  the  simulation  models  is  irrelevant  as  both  essentially  equal  zero. 
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c.  Maintenance  Man  Hours  Per  Flight  Hour 

Regression  is  used  to  fit  a  linear  model  to  explain  the  relationship  between 
the  predictor  variables;  module  reliability,  prognostics  accuracy,  false  positives,  depot 
turn  around  time,  inventory  and  detection  lead  time,  and  the  response  variable  MMHF. 
Figure  27  is  the  JMP  regression  report.  The  parameter  estimates  table  of  the  report  gives 
the  estimated  coefficients,  their  standard  errors  and  the  corresponding  t-statistics  to  test 
the  partial  effect  of  each  variable.  is  0.984,  indicating  the  regression  does  a  good  job 
in  accounting  for  the  variability  in  MMHF.  Module  reliability  is  the  only  predictor 
variable  chosen  for  the  linear  model.  False  positive  is  the  next  significant  variable,  but 
not  significant  enough  to  be  included  in  the  model. 
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R  Square  0.984845 

RSquare  Adj  0.98268 

Root  Mean  Square  Error  0.005854 

Mean  of  Response  0.236853 

Observ  ations  (or  Sum  Wgts)  17 

I  Analysis  of  Variance 
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13.9378 

0.0022 

Figure  27.  JMP  Regression  Report  for  ALS  MMHF 


59 


The  scatter  plot  matrix,  Figure  28,  indicates  there  is  no  collinearity 
between  module  reliability  and  false  positives.  The  plot  shows  a  linear  relationship 
between  module  reliability  and  MMFIF.  This  supports  the  findings  of  the  regression 
model:  0.3  -  (0.00014  *reliability)  -  (6.5e  -  8  *  (reliability  -  505.88)''2) . 


.2.225.25.275.3.325  0  250  500  7501000  45050055060065070075 


Figure  28.  Scatter  Plot  Matrix  ALS  MMHF 
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Figure  29,  is  a  contour  plot,  comparing  module  reliability  and  false 
positives  in  terms  of  MMHF.  The  vertical  coloring  pattern  suggests  failure  rate  improves 
with  the  increase  of  module  reliability  and  is  not  affected  by  false  positives.  This 
provides  further  evidence  in  support  of  the  fitted  regression  model. 


Contour  Plot  for  MMHF 
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Figure  29.  Contour  Plot  ALS  MMFIF 


Figure  30  is  a  plot  of  the  MMHF  linear  model  of  the  ALS  system  over  the 
module  reliability  range.  The  fitted  regression  model  predicts  a  decrease  in  MMHF  when 
module  reliability  increases.  Therefore,  the  largest  value  of  module  reliability  in  the  range 
is  used  in  the  regression  model  to  calculate  the  smallest  MMHF.  The  highest  value  of  the 
module  reliability  is  1000  hrs.  Module  reliability  is  set  to  1000  hrs.  The  regression  model 
produces  an  MMHF  of  0.17785.  The  DBS  model  for  the  ALS  produces  an  MMHF  of 
0.1590  with  standard  error  0.0016.  The  relative  difference  between  the  regression  model 
and  the  simulation  model  is;  1 1 .8%. 
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Linear  Regression  Models  for  ALS  MMHF 
over  increase  in  module 
reliability  range 


0.35  T 
0.3 
0.25 

IL  0.2 

I 

Z 

^  0.15 
0.1 

0.05 

0  ■ 


O  /O 

^  ^  ^  n?  4*  ci?  (b^  (i^  cf 


fcP  V 


Increase  in  Module  Reliability  (hrs) 


Figure  30.  MMHF  Regression  Model  for  ALS 


E.  COMPARING  THE  TRADITIONAL  LOGISTIC  SYSTEM  TO  ALS 

For  FPM  and  MMHF  linear  models  can  be  used  to  compare  the  traditional  system 
and  the  ALS.  For  operational  availability  a  screening  technique  is  used  to  compare  the 
traditional  system  and  the  ALS. 

1,  Comparison  in  Terms  of  Operational  Availability, 

The  data  initially  generated  consisted  of  17  design  points.  For  this  data, 
operational  mean  for  the  traditional  system  is  0.996726  with  a  standard  deviation  of 
0.000566  and  for  the  ALS  is  0.997518  with  a  standard  deviation  of  0.0006777.  The  ALS 
has  the  larger  mean,  but  it  also  has  the  largest  standard  deviation.  The  difference 
between  the  two  systems  is  not  clear. 

Rinott’s  two-stage  screening  method  is  used  to  select  which  logistic  system  is  the 
best.  The  method  is  fully  explained  in  (Chen  &  Kelton,  2000).  To  use  this  method  it  is 
assumed  the  data  sets  are  normally  distributed.  To  satisfy  the  assumption  each  DBS 
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models  was  run  an  additional  thirty  times.  The  thirty  design  points  were  seleeted  using 
NOTH.  The  design  points  and  the  eorresponding  output  are  listed  in  Appendix  G.  In 
total  there  are  46  observations  for  eaeh  system.  Eaeh  data  set  may  be  eonsidered 
normally  distributed  by  applying  the  general  rule  of  thumb;  if  a  data  set  is  greater  than 
thirty  the  Central  Limit  Theorem  (CLT)  ean  be  used  (Devore,  2004,  240).  The  CLT  state 
large  independent  random  samples  (data  sets)  are  approximately  normally  distributed. 

A  spreadsheet  written  by  Professor  Susan  Sanehez,  Naval  Postgraduate  Sehool  is 
used  to  implement  Rinott’s  two-stage  sereening  method.  The  ALS  is  ehosen  as  the  best 
system  after  the  first  stage.  The  seeond  stage  is  not  neeessary. 

2.  Comparison  in  Terms  of  FPM 

For  FPM,  the  regression  indieates  module  reliability  is  the  most  important  faetor 
for  the  traditional  system  and  prognostie  aeeuraey  is  the  most  important  faetor  for  the 
ALS.  Constraints  may  preclude  investing  in  both  module  reliability  and  prognostic 
accuracy. 

The  optimal  (smallest)  value  for  the  FPM  linear  model  of  the  traditional  repair 
system  is  0.012,  when  module  reliability  is  1000  hours.  The  FPM  linear  model  for  the 
ALS  includes  prognostic  accuracy  and  module  reliability.  Since  only  one  variable  can 
change,  module  reliability  is  set  to  a  constant  zero.  This  means  module  reliability  is  not 
improved.  The  optimal  (smallest)  value  is  0.00005  when  prognostic  accuracy  is  1.0.  The 
worst  (largest)  value  is  0.001  when  prognostic  is  0.9.  For  the  parameter  ranges,  the 
prognostic  worst  case  for  the  ALS  yields  a  lower  FPM  then  the  best  case  for  the 
traditional  system.  In  this  case,  investing  in  prognostics  is  clearly  better  than  investing  in 
module  reliability. 

In  the  case  when  one  option  does  not  dominate,  graphs  like  Figure  31  can  be  used 
to  find  the  turning  point.  The  turning  point  is  the  values  for  which  one  option  is  better  or 
worse  than  the  other.  To  construct  Figure  31,  first  set  the  two  regression  models  equal  to 
each  other  and  solve  for  prognostic  accuracy.  The  result  is  an  equation  for  prognostic 
accuracy  in  terms  of  module  reliability.  This  is  the  equilibrium  equation.  The  pair  of 
values  of  prognostic  accuracy  and  module  reliability  that  satisfy  the  equation  yield  the 
same  FPM:  in  terms  of  FPM  they  are  equivalent.  Thus,  investing  in  one  or  the  other  is 
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the  same.  Select  values  above  and  below  the  equilibrium  line,  to  determine  which  option 
is  best.  Note  the  graph  of  Figure  31  has  prognostic  accuracy  between  values  of  0.15  and 
0.45.  This  coincides  with  the  conclusion,  but  the  model  prognostic  accuracy  range  is  0.9 
to  1.0.  Therefore,  for  this  case  Figure  31  should  not  be  used  for  further  analysis.  For  the 
entire  parameter  ranges  the  ALS  performs  better  than  the  traditional  repair  system.  When 
deciding  between  investing  in  module  reliability  in  the  traditional  repair  system  or 
prognostic  accuracy  it  is  best  to  invest  in  prognostics  and  switch  to  ALS. 


Traditional  versus  ALS  FPM 
(increase  in  module  reliability  versus 
prognostic  accuracy) 
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Figure  3 1 .  FPM  Regression  Models,  Traditional  Repair  System  Versus  ALS 


3,  Comparison  in  Terms  of  MMHF 

For  MMHF,  the  regression  indicates  build  window  and  module  reliability  are  the 
most  important  factors  for  the  traditional  system  and  module  reliability  is  the  most 
important  factor  for  the  ALS.  Two  questions  are  answered;  (1)  can  increasing  only  build 
windows  for  the  traditional  logistic  yield  similar  of  better  results  then  the  ALS,  (2)  with 
build  windows  set  at  the  optimal  value  (500  hrs),  is  it  better  to  invest  in  module  reliability 
in  the  traditional  logistics  system  or  the  ALS. 
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First  a  comparison  is  made  between  the  linear  model  of  the  traditional  logistic 
system  over  the  build  window  range  with  no  improvement  in  module  reliability  range  and 
the  ALS  model  over  the  module  reliability  range.  Figure  32  is  a  plot  of  the  equilibrium 
equation.  When  no  improvement  in  the  ALS  module  reliability  is  made  the  two  systems 
are  equivalent  when  build  windows  is  increased  to  over  1000  hours.  The  build  window 
range  is  50  to  500  hours.  Therefore  Figure  32  should  not  be  used  for  further  analysis. 
The  regression  models  show  that  within  the  parameter  ranges  of  the  simulation  increasing 
only  build  windows  in  the  traditional  system  will  yield  a  higher  (worse)  MMFIF  than  the 
ALS. 


Tradtional  versus  ALS  MMHF 
(tradtional  build  windows  versus  ALS  module  reliabilty) 


Figure  32.  MMHF  Regression  Models,  Traditional  Repair  System  Versus  ALS 


Next,  a  comparison  is  made  between  the  linear  model  of  the  traditional  logistic 
system  over  the  module  reliability  range  and  build  window  range  set  to  500  hours  and  the 
ALS  model  over  the  module  reliability  range.  Figure  33  is  a  plot  of  the  equilibrium 
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equation.  When  no  improvement  in  the  ALS  module  reliability  is  made  the  two  systems 
are  equivalent  when  module  reliability  is  increased  by  over  500  hours  for  the  traditional 
system.  Eventually  as  displayed  in  Figure  33  for  the  systems  to  be  equivalent  module 
reliability  for  the  traditional  system  is  increased  by  over  100  hours,  which  exceed  the 
module  reliability  range.  The  regression  models  show  that  the  deciding  between 
investing  in  module  reliability  in  the  traditional  logistics  system  or  in  the  ALS,  it  is  best 
to  invest  in  the  ALS. 


Traditional  versus  ALS  MMHF 

(traditional  module  reliability  with  build  window  =  500  hrs 
versus  ALS  module  reliability) 
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Figure  33.  MMHF  Regression  Models,  Traditional  Repair  System  Versus  ALS 

F.  CONCLUSIONS 

This  chapter  outlined  the  results  of  the  thesis.  Design  points  to  drive  the 
simulations  were  carefully  selected  to  extract  as  much  information  as  possible  from  a 
minimum  number  of  runs.  For  both  systems,  regression  led  to  good  predictive  models  for 
FPM  and  MMHF,  but  not  operational  availability.  Additional  runs  were  made  to 
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implement  Rinott’s  screening  method  to  compare  the  operational  availability  of  both 
systems.  For  each  of  the  MOEs,  the  ALS  is  the  better  system.  The  ALS  dominates  in 
terms  of  FPM  and  MMHF.  However,  large  gains  in  operational  availability  were  not 
realized. 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

The  objective  of  the  thesis  was  to  compare  the  traditional  logistics  system  to  the 
ALS  and  analyze  the  hypothetical  benefits  of  the  ALS.  Key  components  necessary  to 
model  the  system  were  identified.  DBS  models  were  developed  using  Java  and  Simkit. 
The  DBS  models  and  the  regression  models  can  be  used  as  decision  making  tools.  The 
regression  models,  only  apply  within  the  predictor  variable  ranges. 

Conclusions  for  the  traditional  repair  system: 

1.  I-level  inventory  equivalent  to  that  from  table  6  with  a  guaranteed  depot 
turn  around  time  of  40  days  is  all  that  is  required  for  operational 
availability.  At  these  specific  values  increasing  the  I-level  inventory  or 
decreasing  the  depot  turn  around  time  has  a  minimal  effect  on  operational 
availability.  At  these  specific  values,  the  main  driver  for  operational 
availability  is  build  window. 

2.  Module  reliability  is  most  important  in  terms  of  BPM.  Shrinking  build 
windows  at  the  I-level  does  not  improve  BPM.  Shrinking  build  windows 
leads  to  some  parts  being  replaced  early  and  more  often  at  the  I-level. 
Replacing  parts  more  often  does  not  decrease  PPM.  The  parts  are  on  a 
replacement  schedule  at  the  0-level.  With  the  replacement  (high  time) 
schedule  in  place,  replacing  parts  earlier  at  the  I-level  does  not  improve 
PPM. 

3.  Module  reliability  and  build  windows  are  the  two  significant  factors  for 
predicting  MMHP.  They  are  equally  weighted. 

Conclusions  for  the  APS: 

1.  I-level  inventory  equivalent  to  that  from  table  6  with  a  guaranteed  depot 
turn  around  time  of  40  days  is  all  that  is  required  for  operational 
availability.  At  these  specific  values  increasing  the  I-level  inventory  or 
decreasing  the  depot  turn  around  time  has  a  minimal  effect  on  operational 
availability.  At  these  specific  values,  the  main  driver  for  operational 
availability  is  prognostic  accuracy  followed  by  module  reliability. 

2.  Module  reliability  and  prognostic  accuracy  are  the  significant  factors  in 
the  PPM  model.  Module  reliability  is  weighted  more  heavily  then 
prognostic  accuracy. 

3.  Module  reliability  is  the  only  significant  factor  in  predicting  MMHP. 
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For  FPM,  the  regression  models  show  that  when  deeiding  between  investing  in 
module  reliability  in  the  traditional  repair  system  or  prognostie  aeeuraey  it  is  best  to 
invest  in  prognosties  and  switeh  to  ALS.  In  faet,  within  the  parameter  ranges  of  the 
simulation,  the  worst  FPM  rate  for  ALS  was  better  than  the  best  FPM  rate  for  the 
traditional  repair  system. 

The  analysis  shows  that  when  operational  availability  is  the  eriteria  used  to 
distinguish  between  the  two  systems,  the  ALS  is  seleeted  as  the  best.  A  performanee 
based  logistie  eontraet  which  guarantees  delivery  of  components  within  a  certain  time 
frame  makes  the  two  logistic  systems  more  equivalent. 

For  MMHF,  the  regression  models  show  that  when  deciding  between  investing  in 
module  reliability  in  the  traditional  repair  system  or  to  invest  in  module  reliability  in  the 
ALS,  it  is  best  to  switch  to  the  ALS.  The  ALS  with  no  improvements  is  equivalent  to 
increasing  both  build  window  and  module  reliability  to  the  maximum  value  for  the 
traditional  system.  The  ALS  potential  far  exceeds  that  of  the  traditional  logistic  system. 


B,  RECOMMENDATIONS  FOR  FURTHER  STUDY 

Potential  follow-on-studies  include  researching  the  effects  of: 

•  different  module  replacement  schedules  (high  time). 

•  different  flight  schedules. 

•  longer  depot  turn  around  times,  to  gain  an  understanding  of  when  depot 
turn  around  time  impacts  operational  availability. 

•  different  ranges  for  the  predictor  variables. 

Further  enhancements  to  the  DBS  models  may  be  applied: 

•  Include  the  depot  level  in  detail. 

•  Model  actual  individual  maintainers  and  equipment. 

•  As  the  JSF  ALS  is  better  defined,  adjust  the  DBS. 

C.  CONCLUSION  AND  RECOMMENDATION  ABOUT  THE  ALS 

For  each  MOE,  the  ABS  out  performed  the  traditional  logistics  system.  The  ABS 
with  prognostic  accuracy  of  at  least  0.9  and  an  exponential  mean  time  between  false 
positives  of  no  more  than  700  hours  dominates  the  traditional  system  in  terms  of  FPM 
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and  MMHF.  Within  the  scope  of  the  thesis,  the  traditional  system  was  never  found  to  be 
the  better  option.  The  ALS  is  superior  to  the  traditional  repair  system. 
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APPENDIX  A 


DATA  VALUES 


The  following  data  were  obtained  from  Lieutenant  Commander  Sehoeh’s  thesis 
(Sehoeh,  2003): 


l-level  site  chosen:  Pax  River 
Number  of  O-levels:  1 

Number  of  aircraft:  25 


Part 

High  Times  (hrs) 

1 

R 

O-level  Trouble 
shooting  time 

l-level 

Inspection 

Time 

T 

Fan 

1100,2000,2200, 

4000,5350,7175, 

7342,7575,10700 

7 

2.4 

N/A 

N/A 

N/A 

Compressor 

2000,2967,3075, 

3800,4000,4150, 

4717,8117 

3 

2.71 

N/A 

N/A 

N/A 

Combustor 

2000 

4 

1.43 

N/A 

N/A 

N/A 

HPT 

1203,1542,2200, 

2417,2483 

2 

2.27 

N/A 

N/A 

N/A 

LPT 

2000,4000,4242, 

16625,19025 

1 

1.31 

N/A 

N/A 

N/A 

Afterburner 

2000 

2 

1.6 

N/A 

N/A 

N/A 

Engine 

N/A 

8 

3.3 

3.34 

4.5 

0 

I  =  Inventory  levels 

R  =  Install  /  Removal  times  (hrs) 

T  =  O-level  to  l-level  transfer  time  (hrs) 

Table  6 .  Data  V alues 
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APPENDIX  B 


ALS  F18  HORNET  EVENT  GRAPH 
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APPENDIX  C 


ALS  O-LEVEL  EVENT  GRAPH 
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F18  Engine 
Installed 
ByOL 


78 


(Module  not  failed  and  impending 
failure  not  detected) 


APPENDIX  D 


ALS  I  LEVEL  EVENT  GRAPH 
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(  No  Engines  Available  In  GoOd 
Engine  Pool.) 
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APPENDIX  E. 


ALS  D-LEVEL  EVENT  GRAPH 
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APPENDIX  F. 


AGE  REPLACEMENT  POLICY 


The  following  is  the  long  run  average  failure  rate  model  for  the  age  replaeement 
poliey  by  Professor  Patrieia  Jaeobs,  Naval  Postgraduate  Sehool; 

An  age  replaeement  poliey  elass  for  an  item  upon  its  failure  or  upon  it  reaehing 
age  T  whiehever  oeeurs  first: 


Replaeement  items  are  as  good  as  new.  Let  L_  be  the  /"■  lifetime.  Assume 
LjL^,...  are  independent  identieally  distributed  having  a  distribution  funetion  Fwith 
density  /.  Let  denote  the  times  between  sueeessive  failures.  Let  N ^{t)  be  the 

number  of  failures  to  oeeur  during  (0,  t].  {A^(0;t^0}  is  a  possibly  delayed  renewal 

Nit)  1 


eounting  proeess  with  inter-renewal  times  {F. },  1im- 


EVy^ 


with  probability  1. 


To  find  E\YA: 


The  random  variable  F^  is  comprised  of  a  random  number  of  time  periods  of 
length  T  (corresponds  to  replacements  not  associated  with  failures)  plus  a  last  time 
period  for  which  the  distribution  is  that  of  a  failure  conditioned  on  failure  before  age  T. 

Y^  =  NT  +  U 

P{N>k}  =  [\-F{T)X  for  k  =  1,2.- 

P{L<u,L<T} 


P{U  <u}  =  P{L<u\L<T}  = 
=  0  if  u<0 

F(T) 

=  1  if  u>T 


P{L<T} 


Result:  Let  A  be  a  nonnegative  random  variable. 

E{N]  =  j:np{N  =  n}  =j:P{N  >n}=  ZP{A  >  n] 

n=0  n=l  n=0 
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Using  the  result  in  this  example 


E[N]  =  >  n }  =  Z[1  -  F(T)Y  = 

F,(T) 

E[U]  =  ly^^du 
F(T) 

E[YJ  =  E[NT  +U]  =  E[N]T  +  E[U]  =  +  1;  u 


E(T) 


E(T) 


(Jacobs,  2005,  12-13  ) 


STATEMENT:  If  the  lifetime  distribution  is  exponential  then  the  log  run 
average  number  of  failures  under  age  replacement  policy  is  the  same  as  the  long  run 
average  number  of  failures  when  the  item  is  replaced  upon  failure  alone. 


Proof: 


E[YJ  =  E[NT+  U]  =  EAiN]T+E[U]  = 


[l-E(T)]T 

E(T) 


let  a  = 


[1-F(r)]r  Te 


E(T) 


1  1  fT^  /(W) 

letb=lu-— 


du  = 


1 


EiT) 

1  1 


llue’^du 


1  1 


a  +  b  = 


[-Af* -e-l 

\\-e  +  \\ 

-Te  "  +  \X-e-^^V A  _  1 
(l-e")  ^A 


+j; 


/(m) 


E(T) 


du 


but  —  =  expected  time  of  failure  due  to  age  if  lifetime  exponentially  distributed. 


Therefore,  the  statement  is  correct. 
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APPENDIX  G 


ADDITIONAL  SIMULATION  RUNS 


Traditional  Model 


Input 

Output 

Reliability 

Depot 

(hrs) 

Inventory 

Build 

window 

Availability 

(hrs) 

(days) 

(modules) 

(hrs) 

10 

30 

8 

200 

0.995 

60 

40 

7 

150 

0.9948 

100 

20 

8 

300 

0.9952 

10 

30 

1 

350 

0.9952 

80 

10 

7 

400 

0.9954 

100 

20 

4 

500 

0.9955 

70 

30 

2 

400 

0.9953 

100 

35 

10 

350 

0.9953 

50 

10 

2 

400 

0.9953 

90 

15 

9 

150 

0.9948 

80 

30 

11 

100 

0.9947 

70 

40 

6 

450 

0.9955 

30 

35 

4 

500 

0.9955 

20 

35 

8 

450 

0.9955 

100 

40 

4 

250 

0.9951 

70 

25 

5 

250 

0.9951 

20 

35 

3 

150 

0.9948 

30 

20 

11 

100 

0.9947 

40 

20 

7 

50 

0.9945 

80 

25 

11 

500 

0.9956 

40 

40 

5 

300 

0.9952 

60 

30 

10 

300 

0.9952 

50 

25 

6 

450 

0.9954 

40 

40 

11 

350 

0.9953 

90 

15 

2 

300 

0.9952 

30 

15 

9 

500 

0.9955 

90 

25 

3 

100 

0.9947 

10 

20 

6 

400 

0.9954 

50 

10 

10 

250 

0.9951 

Depot  =  depot  turn  around  time 

Table  7.  ALS  additional  runs 
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ALS 

In 

put 

Output 

Depot 

(hrs) 

HI 

False 

Postives 

Lead 

Time 

(days) 

(module) 

(fraction) 

nasH 

nasH 

0 

10 

9 

0.94 

600 

15 

0.9995 

100 

30 

8 

0.97 

650 

30 

0.9966 

600 

40 

7 

0.98 

700 

25 

0.9976 

1000 

20 

8 

0.93 

650 

40 

0.998 

100 

30 

1 

0.94 

650 

20 

0.9966 

800 

10 

7 

0.99 

550 

35 

0.9978 

1000 

20 

4 

0.99 

600 

25 

0.998 

700 

30 

2 

0.95 

500 

40 

0.9977 

1000 

35 

10 

0.96 

500 

40 

0.998 

500 

10 

2 

0.91 

650 

40 

0.9974 

900 

15 

9 

0.99 

550 

20 

0.9979 

800 

30 

11 

0.93 

650 

20 

0.9978 

700 

40 

6 

0.91 

700 

30 

0.9977 

300 

35 

4 

0.92 

600 

35 

0.9971 

200 

35 

8 

1 

600 

35 

0.9968 

1000 

40 

4 

0.92 

650 

25 

0.998 

700 

25 

5 

1 

550 

35 

0.9977 

200 

35 

3 

0.96 

600 

40 

0.9968 

300 

20 

11 

0.98 

650 

30 

0.9971 

400 

20 

7 

0.97 

500 

25 

0.9972 

800 

25 

11 

0.95 

450 

20 

0.9978 

400 

40 

5 

1 

650 

20 

0.9973 

600 

30 

10 

1 

550 

35 

0.9976 

500 

25 

6 

0.99 

650 

25 

0.9974 

400 

40 

11 

0.91 

600 

20 

0.9972 

900 

15 

2 

0.95 

650 

30 

0.9979 

300 

15 

9 

0.93 

500 

30 

0.9971 

900 

25 

3 

0.92 

550 

30 

0.9978 

100 

20 

6 

0.98 

500 

35 

0.9966 

500 

10 

10 

0.94 

500 

20 

0.9974 

F=  False  positive  rate 

Depot  =  Depot  turn  around  time 

Lead  =  Detection  Lead  Time 

Table  8.  ALS  additional  runs 
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